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Preface 



In December 1983, Congress mandated a national assessment of Ghaptef 1. The 
requirement, included in the technical Amendments to the Education Cbnsblidatibh and 
Improvement Act (EGiA) of 1981, directed the National Institute of Education (NIE)^ 
to conduct independent studies and analyses of compensatory education programs 
funded under Chapter 1 of EGIA, and to report its findings to Congress by January 
1987. The studies and analyses were to address the following topics with respect to 
Chapter 1: 

m services delivered; 

m recipients of services; 

• background and training of teachers and staff; 
m allocation of funds (to school sites); 

• coordination with other programs; 

• effectiveness of programs on s^tudents' basic and higher order academic 
skills, school attendance, and future education; and 

• a national profile of the way in which local edu national agencies 
implement activities described under Section 556(b) of Chapter 1. 

The mandate also required consultation with relevant members of the House and Senate 

education committees, the requirement for the National Assessment of Chapter 1 is 

reproduced in Appendix A, and a report of its administrative status Is contained in 

Appendix B. 

This is the second evaluation of the Federal compensatory education program that 
Congress has requested. The Education Amendments of 1974 contained the mandate for 
the previous study. Findings from the resulting Compensatory Education Study, which 

_^On October 7, 1985, the NIE became part of the Office of Educational Research 
and Improvement (OERI) within the U.S. Department of Education (ED). 
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was also conducted by the NIE, contributed to the formulation of the 1978 
reauthorization of Chapter Ts predecessor. Title I of the Elementary and Secondary 
Education Act (ESEA). 

Two significant legislative changes have occurred since 1978. First, in 1981 
Chapter 1 of ECIA replaced Title I of ESEA. Chapter 1 retains the purposes of Title I 
but changed certain administrative features of the program. Second^ in 1983 technical 
amendments to ECIA were enacted in an effort to clarify ambiguities in Chapter 1 and 
to restore some Title I provisions that had been dropped or changed in Chapter 1. 
Among these technical amendments, as previously noted, was the requirement for this 
National Assessment of Chapter 1. 

Congress asked that the National Assessment of Chapter 1 provide two interim 
reports in addition to a final report. The first of the interim reports (Kennedy, Jung, 
and Orland, 1986) describes the population of students that Chapter 1 is intended to 
serve-educationally deprived students residing in areas with high concentrations of 
children from low-income families. This second report reviews and synthesizes 
evidence regarding the effectiveness of Title I and Chapter 1 programs. Both interim 
reports draw mainly from data collected in earlier studies or data collection activities. 
The two interim reports are intended to provide policy makers with a broad perspective 
from which to view current Chapter 1 programs, which in turn will be described more 
fully in the third and final report of the National Assessment of Chapter 1. 

The third report will draw from the scries of studies commissioned specifically for 
the National Assessment of Chapter 1. These studies arc described in Appendix C. 
The third report will describe: 

9 The characteristics of Chapter 1 participants; 

• The quantity and characteristics of services being provided by 
Chapter 1; 

• Program administration at each level of educational governance; 
and 
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• How and why districts make decisions about the selection of 

schools and students, the isllocatibn of fuiids among schools, and 
the design of their Chapter 1 programs. 

A particular emphasis of the report will be to identify program practices that have 
changed or remained the same in the shift from Title I to Chapter 1, 

the Chapter 1 Study team began to implement the National Assessment in the 
Fall of 1984, after its Study Plan had been reviewed by Congressional staff members in 
both the Senate and House education committees. Responsibilities for the several 
components of the National! Assessment are distributed among members of the Study 
Team. Mary Kennedy, Richard Jung, and Martin Orland had primary responsibility for 
the first interim report. Mary Kennedy and Randy Demaline took the lead in the 
second interim report. Beatrice Birman, who took over the duties of Director in May 
1986, oversaw the completion of the second interim report and is directing vsrork on the 
final report. Sections within the final report are distributed as follows: Richard Jung 
is responsible for describing the characteristics of program recipients and patterns of 
their participation, Gilbert Garcia for describing Services, Martin Orland for analyzing 
administrative practices, and Ron Anson for describing district-level decisions about the 
program. Paige Russ and Saunders Freeland had primary responsibility for typing this 
report. 



Beatrice F. Birman, Director 

National Assessment of the Chapter i Program 



Ron Anson^ Deputy Director 

National Assessment of the Chapter ] Program 
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Executive Summary 

This report reviews findings from several large national data collection efforts 
designed to estimate the effect of Title I and Chapter 1 on student achievement. The 
studies and data sources that have been used extensively are described in Appendix D. 
While each study cited in this report presents its own problems and strengths with 
respect to its choice of outcome measures and comparison groups, the studies together 
suggest several broad conclusions about the effectiveness of Chapter 1. These 
conclusions are summarized below and discussed in the remaining chapters of the 
report. 

^^^^'^r ' describes Chapter 1 programs and the methods typically used to assess 
their effects. 

^ ^ '^iews evidence regarding the population of disadvantaged children in 
general and how their achievement differs from that of the population of all children. 
The primary finding presented in this chapter is: 

i. TJie achieveSent of disadvantaged students has improved since 
1965. especially in rsading, relative to the achievement of the 
general population. 

mm^r 3 . describes the one-year effects of Chapter 1 -funded programs on 
students' achievement test scores and describes the differences found across grade 
levels and between reading and mathematics. Its main findings are: 

2. Students rccciying Chapter 1 services experience larger increases in 
their standardized achievement test scores than comparable students 
who do not. However, their ^ains do not move them substantially 
toward the achievemisnt levels of more advantaged students. 

3. Students participating in Chapter 1 matheSatics programs gain more 
than those participating in Chapter 1 reading programs. 

4. Students in early elementary Chapter 1 pro^rais gain more than 
students participating in later-grade programs. 
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5. Evidence regarding program effects on student attitudes toward 
sciiool is inconclusive. 

6. Researchers liave not yet developed adequate methods for determining 
the relationship between program costs and program effects on 
standardized achievement scores. 

Chapter 4 looks beyond a single school year to assess longer-term program 

effects. Its main findings are: 

7. The achievement gap between disadvantaged and advantaged students 
appears to widen during the summer months. 

8. Title I-suppbrted summer programs have hot narrowed the gap; however, 
these programs often were hot designed to be academically rigorous.^ 

9. Students who discontinue Title I appear gradually to lose the gains they 
made when receiving services. 

10. Ch^aptcr 1 students wit^^ low_achievement scores appear to maintain 
their relative academic positions but not to move ahead^ However, the 
evidence suggests they would have lost ground relative to their peers if they 
had not received services. 

1!. No nationally-represehtative studies have examined the long-^rm effect of 
Chapter 1 programs on graduation rates, future education, or adult literacy. 

C4iaptef^ reviews evidence regarding the advantages and disadvantages of 

particular project characteristics. Its main fihdihgs are: 

12. Large-scale studies designed to identify particular project 
characteristics that improve student achievement test scores have 
yielded ihcbnsistent or ihconclusive results. 

13. Researchers have recently identified a number of ihstructibhal practices that 
are likely to increase the achievement of disadvantaged students and that 
cah be used ih Chapter 1 programs. 

These 13 statements must be interpreted as indicating general trends, which will 

hot necessarily apply to particular projects, schools, or children. Chapter 1 projects 

vary considerably across states, districts, and even schools within districts. 

Cohscquehtly, ho single statement will apply equally well to ail activities supported by 

Chapter 1. Our findings are also somewhat qualified by the age of the data on which 



^Thcse sunamary statements refer to Chapter ! except when they are based solely 
on information collected prior to the enactment of ECIA. 
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they are based; The most recent data we use to describe achievement gains of 
Chapter 1 students were reported by state educational agencies for the 1983-84 school 
year. In addition to these data, we draw heavily on data collected between 1976 and 
1979. The use of these earlier data to review the effectiveness of Chapter 1 assumes 
the program has not changed enough to alter substantially the conclusions that would 
be reached if a new study were to be undertaken today. Indeed, the authorizing 
legislation for compensatory education has been generally consistent in its structure 
and purposes for over 20 years and Chapter 1 programs have become stable entities in 
many districts. Our final report will provide Congress with details about the current 
operation of Chapter 1 programs and the extent to which they appear to have changed 
over times. 
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1. Introduction 

In 1965, Congress authorized Federal support for compensatory education through 
Title I of the Elementary and Secondary Education Act (ESEA). Since then, there have 
been numerous studies and reviews of the effectiveness of programs supported by that 
legislation. The conclusions from these studies and reviews have varied, depending on 
when, where, and how the studies were conducted. This report examines and 
summarizes existing data on the effects on students' academic achievement of ESEA 
title I and its successor^ ECIA Chapter 1. Appendix D describes the studies and data 
bases most heavily relied upon for this report. 

Chapter 1 Services 

The effects of Chapter 1 have been difficult to assess at the national level, in 
part because Chapter 1 does liot require a particular instructional program. Instead, it 
permits districts to design programs they believe will promote the educational 
development of their particular population of students. Consequently, districts and 
even schools within districts differ in the grade levels they serve, the procedures they 
use to select students, the services they provide, and the administrative strategies they 
use to orchestrate those services. 

Figure 1.1 indicates that 75 percent of the participating students receive 
instructional services in reading, and that nearly half of all participants receive 
instruction in mathematics. Chapter 1 students also receive instruction in language 
arts and in special subjects such as English as a second language, and they receive 
support services such as attendance, guidance, health, and nutrition assistance. 
Figure 1.1 docs not indicate, however, the variety of services that may be provided 
within a single service category such as reading or mathematics. A student may 
receive Chapter 1 reading services, for instance, for 20 minutes three days a week or 
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Figure 1.1 
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Figure reads: Seventy five percent of ail students Enrolled In Chapter [ programs received supplementary 
reading instrUctiph during the 1983-84 school year; 46 percent received supplementary 
mathematics instruction. 

nptal hurtlberof stud_e rnillibh. If children receive mbrB than one service, they are obUhted in each 

subject in which they are enrrDiied. 

^'Tncludes vocational instruction and special services for handicapped students. 
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for two hours every day.^ These services may be provided in the child's regular 
classroom or in another setting; they may be provided by teachers^ teacher 
aides, reading or math specialists or parents; and they may reiy on the satae 
instructional materials as the regular classroom or use difjferent materials. Even when 
services are all labeled "reading," the particular reading skills emphasized can Vary 
dramaticaliy. 

Measurinir^the Effects of Chanter 1 Services 

Studies of Chapter 1 have usually responded to this diversity by limiting their 
scope to reading and mathematics programs. Such a focus results in more manageable 
evaluations, and covers the most commonly provided services. But it also precludes 
evaluation of the impact of other Chapter 1 services. 

Studies of Chapter 1 have also limited their own scope by measuring the impact 
of Chapter 1 programs mainly with standardized achievement tests. These tests cover 
the range of academic content taught to the general student population. They are not 
intended to nieasure the specific content taught by any one instructional program. 
Consequently, they will measure achievement in areas that were never intended to be 
taught, and wilt fail to document some of the skills that students learn as a result of 
their participation in a program. This mismatch between the content of the test and 
the content of the curriculum is unavoidable, because standardized tests are not 
designed to measure changes attributable to speciific programs but instead to provide 
school districts and others with a general sense of what students know, relative to the 
population in general. Yet because of this content mismatch^ when standardized 
achievement tests are used to measure the growth of a particular student and to 

^Services provided for more than 25 percent of the school day require a 
contribution of resources from the school district. 
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attribute that growth to a particular program^ the tests will generally uhderestiinate 
the amount of program-related growth students have actually experienced during the 
year. 

Some educators have argued that student outcomes such as attitude toward school, 
motivation, and high school graduation are more meaningful indicators of a program's 
success thai! are changes in achievement test scores. Indeed, the long-term 
consequences of changes in achievement test scores is not always clear. Thus, while 
they arc valuable indicators of student learnings achievement test scores do not reflect 
the long-term consequences of Chapter 1 program participation, arid they only 
imperfectly measure the basic reading and mathematics knowledge that they are 
designed to assess. 

At4ri%wtinch AiAieveffl^^;^^ 1 Projects 

Several strategies have been used to estimate the impact of Chapter 1 services. 
One is to compare the achievemerit of disadvantaged students to the achievement of 
advantaged students, and to determine whether the gap between these two groups is 
reduced over time. Some analysts have used data from the Natiorial Assessment of 
Educational Progress in this way, and their findings arc reviewed in this report. This 
strategy does not take into account which students actually receive program services 
and which do not. Consequently, it cannot determine the extent to which Chapter 1- 
funded programs contributed to any observed population changes. 

it would be possible to alter these comparisons to include only disadvantaged 
students who actually have participated in the program, and detefmirie whether the gap 
between this group and the population of advaritaged studerits changes as participants 
receive services. This approach would describe the extent to which the achievement 
levels of Chapter 1 participants came to resemble that of their more advantaged peers. 
But because advantaged aad disadvantaged students may have different learning rates 

4 

EKLC 



aiiyway, this apprbacfi would not indicate how much larger or smaiier the observed 
changes were than would have been expected if the disadvantaged students had not 
received Ghaptef 1 services. 

The strategy preferred by evaluators is to compare the achievement of students 
receiving Ghaptcr 1 services to comparable disadvantaged students not receiving 
Chapter 1 services. But this approach is difficult to implement. It requires the 
cvaluator to identify and test a group of students who do hot receive Chapter 1 
services, but who are comparable to those who do. Yet if Chapter 1 funds are 
distributed appropriately, there should be very few schools or students comparable to 
those receiving Chapter 1 services within a given school district. Chapter 1 requires 
school systems to provide their Chapter 1 services in schools with the greatest need 
and to students within those schools who are themselves among those most in need. If 
evaluators select comparison groups from within Chapter 1 schools, they generally 
select students who achieve at higher levels than the Chapter 1 participants. Oh the 
other hand, if evaluators select cbmparisoh students from schools that do not offer 
Chapter 1 programs, they may find students with comparable achievement scores but 
with superior educational environments.^ the more the comparison group differs from 
the group of students receiving Chapter 1 services, the more difficult it is to use its 
achievement as a benchmark for estimating the effects of Chapter 1. Despite the 
difficulty of implementing this approach, it has the advantage of estimating the 
achievement growth that Chapter 1 participants would have demonstrated if they had 
not been served. 



In our first report to Congress, we showed chat student achievement was_ 
adversely affected by attending schools with high concentrations of poverty. Generally 
speaking^ hon-Chaptcr 1 schools have unusually low concentrations of poverty^ ahd 
consequently are likely to provide more beneficial learning environments for students. 
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Pi^fOTiOutcdines Not Addressed iii-Htis^ePOTt 

The analyses examined in this report tell us a great deal about the impact 
Chapter 1 has had on student achievement. But they do not address all of the student 
outcomes of interest to Congress. Iii its mandate for this studyi Congress asked for 
information about "... effectiveness of programs on students' basic and higher order 
academic skills^ school attendance, and future education . . .** Because this report used 
only existing data and because these data were mainly standardized achievement test 
scores gathered within a year or two of students' receipt of Chapter 1 services, the 
report does not address three of Congress' concerns. First, standardized tests do not 
separately measure higher-order skills such as written composition and abstract problem 
solving and consequently do not permit us to distinguish the potential contributions 
Chapter 1 may have made in these areas. Second, ho national data base contains 
evidence regarding the school attehdahce of Chapter 1 students. And finally, no 
national study has followed students for more than two years after they leave 
Chapter 1, so that it is not possible to ascertain the program's impact on students' 
future education. 

There is some evidence that students' outcomes in such areas as school 
attehdahce and future education differ from their test score imprdvemehts. In studies 
of early childhood programs (Smith, 1985), student test scores rose while students 
received services and later declined, yet other Indicators of educational success 
continued to show evidence of program benefit. Such a pattern may not appear for 
Charter 1 students^ since Chapter 1 services arc generally more limited in scope than 
arc preschool services. Nevertheless, these patterns serve as a remihder that, while 
the achievement data presented here are important^ they do not present a complete 
picture of the potential effects associated with Chapter 1 programs. 
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2. trends In the Achievement of Disadvantaged Children 

Gentrai Finding 

. 1. The achicvcmcht of disadvantaged students has improved since 
1965. especially in reading, relative to the achievement of the 
general population. 

This chapter reviews evidence regarding achievement trends in the population of 
school-age children. It contrasts changes that have occurred over time among 
different subgroups of the population, with particular emphasis oh those groups most 
likely to have received Chapter 1 services. 

Chapter 1 is designed to support services specifically for low-achieving students 
who attend schools with high concentrations of children from poor families. The 
legislation provides extensive guidance regarding the selection of both schools and 
students to participate in local pirbgrams. Approximately 5 miliion students participate 
in Chapter 1 each year^ about 1 1 percent of the entire student body (Carpenter and 
Hopper, 1985). Of these, approximately 40 percent arc new to the program each year, 
while a comparable group who received services the preceding year are removed from 
the program (Carter, 1984). Because a slightly different group receives services each 
year, the proportion receiving services all together is larger than the proportion 
receivii^g services at any one time. By the time students in some districts reach 
fourth grade, for Instance, an estimated 25 percent have received services at some time 
during their school careers.^ In districts and schools that serve larger proportions of 
students within each school year, the total percent served over time would be even 
larger. Given the legislated rules regarding participation, it is reasonable to suppose 
that these students represent the most educationally-deprived students within each 



^This estimate is based on multi-year participation data in two districts— 
St. Louis, Missouri and Lincoln, Nebraska. For details^ see Pfannensteil, 1986. 
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district' 



The amount of services these students receive is also highly variable. A small 
fraction of these fourth grade children have received Chapter 1 services daily for the 
entire five-year period; others have received services throughout one or two school 
years, and still others received a few months or even just a few weeKs of special 
assistance. Furthermore, all of these students receivedi throughout the entire five-year 
period, their regular school program. If we were to examine the entire population of 
fourth graders, the particular contribution Chapter 1 has made to their educational 
achievement as a group would be extremely difficult to define, in part because the 
nature and extent of their participation in Chapter 1 would have been so variable, and 
in part because it would not be possible to separate the benefits of these services from 
the benefits of their regular full-time educational program. 

Without khowinjg which students received services, or when cr for how many 
years they received services, the most general way to examine Chapter I's effects on 
student achievement is to examine changes in the achievement of all disadvantaged 
students^ and to overlook considerations of the nature or extent of their actual 
participation in the program. Such an examination cannot indicate the specific effects 
of Chapter 1, of course, but it can indicate important trends in the achievement of the 
population that Chapter 1 is intended to serve. 



Achievement T^-efHte^PiH4ng t^ 197Ds 



The National Assessment of Educational Progress (NAEP) gathers nationally- 
representative achievement data on 9, 13, and 17 year-old students. Although NAEP 
does not identify all categories of students who might be designated as educationally 
deprived, it does provide several groupings relevant to bur analytic purposes. These 



^In our first report to Congress, however, we indicated that many participating 

schools do not serve particularly large proportions of low-income children. 
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include (1) low-achieving students^ defined as those in NAEP's lowest achievement 
quartiie, (2) students attending schcois located in disadvantaged urban communities; and 
(3) minority students, the first two of these groups are the specific focus of 
Chapter Ts student and school selection procedures and its funding formula. The third 
is not specifically targeted; however participation data indicate that a relatively large 
proportion of minorities do participate in local Chapter 1 programs. 

In 1984, the director of NAEP delivered testimony to Congress regarding changes 
in the academic progress of many cf these students (LaPointe, 1984). His findings are 
presented in Table 2.1. The table entries indicate changes over time in the average 
percent of test items that students answered correctly. Asterisks mark the entries 
that are statistically significant. 

Table 2.1 indicates several important patterns. First, imprbvcihcnts were larger 
among students in the lowest quartilc (the lowest-achieving 25 percent of the total 
student population) than among those in the highest quartile^ especially in reading. 
Second, black students tended to demonstrate more and larger changes than white 
students. Third, 9-ye3r-olds gained more than 13- and 17-year-olds, especially in 
reading and mathematics; and finally, students improved more in reading and 
mathematics than in science.^ 

Not all of these patterns have been corroborated in other data bases. However, 
using these data, LaPointe (1984) and Riddle (1984) have argued that the patterns of 
increases in NAEP achievement data parallel patterns of Federal financial support to 
schools. For instance, most Federal funds support programs for younger students, and 



^ ^Table 2.1 also shows some important patterns of deereases over time. For 

i^ns^tance, the top quartile of white students at all age levels decreased their percent 
correct in mathematics and science items during this period of time. This is clearly a 
pattern worth attending to, though its relationship to Chapter 1 services is unclear. 

9 



2Q 



table 2:1 



Change in Percent of Correct Items by 
Subjec t, Race, Age, and Achievement Quartile During the 197ds,** NAEP 



Bjack Studehts white Students 



Ciuartile 


Age 


Reading 


Science 


Matliematics 


Reading 


Science 


l\^athematics 




9 


a4* 


-0.7 


2.9* 


4.«* 


i.r 


-05 


Lowest 25% 
of Students 


13 


35* 


13 


2.6* 


15* 


2.0* 


03 




17 


1.1 


-05 


1j8* 


-1.7* 


0.7 


-13' 




9 


3.0* 


1.1 


2.6* 


1;2 


-2.4* 


-3.3* 


Highest 25% 
of Students 


13 


25* 


-05 


-25* 


d4 


-4.1* 


-a2* 




17 


-1.1 


-3.9* 


-55 


-03 


-4.2* 


-4.3* 



Table leads: Among studfntejn the 1^ percent of reading items answered oorrectly by 

9-year-old black students increased 8.4 percentag©j3olnte. The percent of readincL items answered 
correctly by 9-year-old white students in the sarne group Increased 4.6 percentage points. 



•Denotes figures that are statistfcally significant at the 5 percent confidence level — i.e.. given the range of 
scores and the size of the relevant fx)pulatibnsi there is a probability of less than 5 percent that the indicated 
change in aveiBge soores has occurred simply by chance. 

**The specific time intervals. vary^ with the subject. In reading, the test points were 1970, 1974, and 1979. In 
mathematics, they were 1972i ig77i and 1981. In science; they were 1969, 19^, and 1976. 



Source: W. Riddle, Ac^lm^mm Score_ tmds and i^&Bi Jnvolverrm Bementary and Sexx^ndEoy ma<xtton: An B(pk>mtbn ot Their 
Relathnshfps, Report 84-627EPW. Washington, aC: Congressional Research Service, The Ubrary of Congreiss, May 3, 1984. 
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this Was the age group showing the largest gains. Second, Federal funds focus on low- 
achieving students, which is the population for whom the greatest improvements were 
observed. Third, Federal funds provide services to higher proportions of minority than 
nonminority students, and minority students have shown greater achievement 
improvements than nonminority students. Finally, Federally-supported services focus on 
core subjects such as reading and mathematics, and the achievement Increases have 
occurred in these areas rather than in science. According to both authors, the general 
pattern of test performance changes over time seems to parallel the pattern of Federal 
education investments. However, both authors discussed Federal programs in generaU 
rather than Chapter 1 in particular, and Federal investments m other programs were at 
their peak during the period covered by these analyses. Furthermore, aiy number of 
other social changes that occurred during this period of time could have influenced 
these patterns. Consequently, it is difficult to estimate the extent to which the 
patterns of improvement indicated by NAEP derive from the particular children who 
received Chapter I services. 

Mdehce of the Ac hievement nf 



on 



Disadva nta ged Students 

Since LaPointe and Riddle presented their data in 1984, two new sources of 
evidence on achievement trends have become available: (1) a new NAEP report 
reading achievement summarized data from 1971 to 1984 (NAEP, 1985), and (2) a report 
by the congressional Budget Office (CBO) summarized achievement trends fouSd in a 
number of testing programs (Koretz, 1986). 

Figure 2,1 presents the recent NAEP findings regarding reading achievemeSt 
among black, Hispanic, and white students. All three groups show evidence of 
improvement u'uring this time interval, but black and Hispanic students started at much 
lower levels of achievement and showed much greater change over time. For instance, 

ii 
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Rgiire 2.1 



liieindsJh Average NAEP Readihg Proficiency Scores* 
for Black, Hispanic, and White Students 



Reading Proficiency Scores 
320 .— 
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200 



180 



160 — 



Age 17 



Age 13 



Age 17 



Age 17 



Age 13 



Age 13 



Age 9 



Age 9 



Age 9 



-J 



1971 1975 1980 1984 

Black 



1971 1975 1980 

Hispanic 



1984 



1971 1975 I960 

White 



1984 



Figure reads: The average raading score of blac?-: 9 year olds was apprbximSely 169 Jn 1971 md 188jri 1984, a 
gain of 19 pointe. the average score of white 9 year olds was 214 in 1971 and 220 in 1984. a 
gain of 6 points. 

•These scores are deri\^ from item response theory ^ased on a scale thS ranges ^rn 0-500, ihese scores 
pj^jte Gsmmm 8(^\e on which compsHMhs coi be mluja age arid test groups. Sooks on the 

srale relats p five levels of proficiency: rudimentary (15(9, t)asic (200), intemiediate (250), adept (SCXQ, and 
advanced (350); 



Soan»: National Assessment of Educational Piogr^, ThB maOng^mpsnxim, Pmgtws Ttmnl ExaeHenca In our Schools: Ttends In Reading 
Over nsv NetksnBl ^mmnem, 1sn-19B4. Prinoaton: Ediicationai Te^rig Service, 1^ 
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black 9.year-5ldi scored around 20 points higficf in 1980 and 1984 than they did in 
1971. The reading performance of black students in 1984 still lags far behind white 
students, but the size of the gap has been reduced considerably in comparison to the 
gap in 1971. The patterns of minority achievement found in 19^4, then, are still 
consistent with those originally observed by LaPointe. 

these NAEP analyses do not examine changes in each achievement quartile and so 
cannot be compared to the earlier findings on that dimension. However, they do 
indicate achievement trends among students living in various types of communities. 
The NAEP findings oa this dimension are shown in Figure 2.2. This figure indicates 
that achievement has improved among students residing in rural and disadvantaged 
urban areas more than it has in advantaged urban areas.^ with the most substantial 
improvements among 9-year-olds residing in disadvantaged urban areas. 

Interest in these and other achievement patterns prompted the CBO to examine 
achievement trends evident in NAEP and in other data bases (Koretz, 1986). The CBO 
analysis was primarily concerned with the overall national decline and subsequent 
upturn in student achievement test scores. It enables us to place the NAEP 
achievement patterns in a larger context. CBO found, for instance, that while 
achievement scores on a wide variety of tests had been declining through the decades 
of the 1960s and 1970s, the trend actually began to reverse itself with children born in 
approximately 1963. These children entered school in about 1968. three years after the 
passage of the EieSentary and Secondary Education Act of 1965. and were nine years 
old in 1972. CBO also found that the decline was most severe among older 



;nc.,«l!'^H- *5"'''" residential areas according to employment statistics For 
instance, a disadvantaged urban area is one with a population of at leas 260 000 a nd 
^^r^TV^^"" proportion of managerial aL profession^ workerf Sal areas 
digged '^mUr'^"°" '"^^ ^"^'"^^ - - 
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Figure 2.2 

Tliehcis in Average NAEP Reading Proficiency Scores* lype of Gommunity 



Reading Proficiency Scares 
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J 
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Figure reads: 1lie ajutage reading proficiency ^re j^^^ yeiEU* olds Jn jrui|d 201 
in J971 and 206 in 1984, a 5 poim increase. In disadvantaged urban communities, 9 year olds 
scx)red 178 in 1971 and 194 jn 1984, a 16 point gain. In advaiitagad urban settings, 9 year olds 
scored 231 in tx>th 1971 and 1984. 

*These scotss are derived from item response theory. Based on a scale that ranges from D-SCQ, these scores 
provide a commpn s<^^^ which jssrnppis^^^ be ma^e for dlff(3rertt age artd teJ^ groupKS. Scares on the 
scale equate with five proficiency levels: rudimenteuy (150), basic (200), intennecliate (250), adept (300), and 
advanced (350). 



Source: National Auessment of muC9^kxwLProgmB,JheGBa(llng_f^^H^ Pa^imss "Rmod Mxm^snce in our Softool? "Bwids In flBmtng 
OmfouttMona/^^mmm im-mi. Prinoeton: Edositional IMng Servioe, 198a 
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students and least severe among younger students. Iii fact it found that scores of 
students in the upper elementary grades are at their highest level in three decades; 
Third, CBO found the decline was most severe on measures that required students to 
use higher-ordeif thinking skills such as problem solving or drawing inferences, and 
least severe on measures that require more basic skills such as recalling facts or 
computing. Finally, CBO found that the general decline in achievement test scores was 
less apparent among minority students than among other groups, while the upturn was 
particularly strong among minority students, especially those in elementary and early 
secondary grades. The CBO analysis did not find, however, clear evidence of 
differences in improvements between low-achieving and high-achieving students, and it 
did not find clear differences in the long-term trends of reading and mathematics 
versus science. 

Summary and Dij 



The central finding reported in this chapter is: 

1- Th$ . ^glH<gvemcnt of^jsadvant a ged students has impr oved since 
J g0gi^SPCciallv in reading, r e lative to the achievement^ t4»e 
gcneFal^pemHatioa. 

Although analyses of population trends cannot be used to infer direct 
programmatic effects, such analyses provide important information about the 
achievement of students who are intended to benefit from Chapter 1 services. As we 
have seen, it is possible that a substantia! portion of educationally-deprived children 
have received some amount of Chapter 1 services at some point in their educational 
careers. Their participation in Chapter 1 could have contributed to the observed 
improvements in the educational progress of disadvantaged and minority students. 

On the other hand, many events in the past decade and a half could account for 
the changes reported by NAEP and CBO. School districts have tried to desegregate 
their schools; State and local educational agencies have increased their focus on basic 

15 



26 



skills and their overall spending on educational services for disadvantaged students; 

minority groups have become more self conscious arid politically active; and teachers 

have becoriie more aware of thei? obiigations to iow-achievirig and minority students, 
the most substantial improvements during this period occurred among black and 

Hispanic students, regardless of residence, and among students who reside in 
disadvantaged Urban areas, regardless of their race or ethnicity. Thus, trends in 
student achievement suggest that the nation is improving its education of disadvantaged 
students. To identify and measure the specific impact of Chapter 1 programs oh 
achievement, we turn now to a review of available national data contrasting students 
who actually participated in Chapter 1 programs with comparable students who did not. 
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3. Compensatory Education and Achievement Test Scores 



eehtra! Findings 



Program Effects on St u dent Ach ievement 

2. Students receiving Chapter 1 services experience larger increases 
in their standardized achievement test scores than comparable 
students who do not. However, their gains do not move them 
substantially toward the achievement levels of more advantaged 
students. 

3. Students participating in Chapter 1 mathematics programs gain more 
than those participating in Chapter 1 reading programs. 

4. Students in early elementary Chapter 1 programs gain more than 
students participating in later-grade programs. 

Pix>^fam^f^cts Qn^^hftr Aspc^^^ of Student Development 

5. Evidence regarding program effects on students' attitude toward 
school is inconclusive. 

Costs Relative to Effects 

6. Researchers have not yet deve adequate methods for determining 
the rciationship between pro costs and program effects on 
standardized achievement scores. 

This chapter focuses not on the population of disadvantaged children in general 

but on students who actually receive Chapter 1 services. It presents evidence 

describing the average effects of Chapter 1 on participating students and how these 

effects vary across grade levels and subject areas. The chapter also examines briefly 

Chapter 1 effects on nonacadcmic areas of student development and the relationship 

between program costs and program effects. 
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Prottrafh-EfT^^fhS t4iaeilt Ac4i^^^^ 

To address the question of whether Chapter 1 programs increase students' 
educational achievement, we rely primarily oh data from two sources: the 
Title I/Chaptcr 1 Evaluation and Reporting System (TIERS) and the Sustaining Effects 
Study, 

TIERS AclM^vcment Pata 

Developed in the mid-1970s, TIERS provided a framework within which State and 
local educational agencies would be required to report comparable data to the U.S. 
Department of Education regarding their Chapter 1 projects and the students they 
serve. The system included standardized procedures for school districts to use iii 
reporting the number of students receiving various Chapter 1 services and in measuring 
the impact of these services on student achievement. TIERS permitted the aggregation 
of project data to State and then to national levels, thus enabling the development of 
a national summary of achievement test scores for Chapter 1 participants. The ECIA, 
enacted in 1981, eliminated the requirement that State and local educational agencies 
implement these or any other standard evaluation procedures, though many State and 
local agencies continue voluntarily to use the TIERS procedures^ 

Though Chapter 1 services arc offered to students ranging from prc-kindcrgartcn 
through twelfth grade^ the services are concentrated in the elementary grades, with 
nearly 70 percent of Chapter I participants in grades one through six. Figure 3.1 
displays the percentage of Chapter 1 students who receive services in each grade level, 
pre-K through 12. It indicates that Chapter 1 students are roughly equally distributed 
among grades one through six, and that progressively fewer students are served in each 
successive grade. 



^Loca) projects arc still required to evaluate their programs' impact on students 
annually, but they need not use standardized procedures to do so. 
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Percent of Chapter I Stude nts R eceiving Services by Grade, 1983-84 



l^rcent of Chapter I Students'" 




Grade 



Figure reads: One percent of all Chapter j students served in the 19^-84 school year were enrolled in pre- 
kindergarten and six percent were enrolled in kindergarten. 

^otal number of students 4S5 million 



I. A. Carpenter and R A. Hopper, Syntttesis of Chapter / Data; Summary Report. Restan, VA: Advanced Technology, 198& 
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Figure 3.2 summarizes the average achievement levels of students tested under 
TIERS in 1983-84 for reading and mathematics:^ the slanting lines in Figure 3.2 
represent the change in average scores, expressed as percentile ranks, for students 
receiving Chapter 1 services during the 1983-84 school year in grades two through 
twelve.® A percentile rank is a form of test score indicating the percent of all 
students nationwide who scored below that achievement level. The average student 
nationwide would achieve at the 50th percentile rank at the beginning of the year and 
at the end of the year as well. In Figure 3.2, the average second-grade Chapter 1 
student performed at the 29th percentile rank in reading in spring 1983^°; higher than 



29 percent of all second-grade students tested in national norming samples. In spring 
1984, after having received reading services supported by Chapter 1, this average 
Chapter 1 student's score was at the 31st percentile, two percentile ranks higher than 
a year before. Thus, the average second grader participating in Chapter 1 gained 
enough during the school year to surpass two percent of the nation's second-grade 



students by the end of the school year. 



^Some States reported for only a few grades or a few districts, and some sub- 
mitted no report. It is difficult to know the extent to which State decisions to submit 
or not to submit particular data elements influence the validity of national averages. 
However, we do know that TIERS data show Very similar national patterns from year 
to year, even though State participation fluctuates and State-specific patterns fluctuate. 

^TIERS does not provide for the collection of achievement data from first-grade 

stu^dents, because it is not feasible to test these students in the spring preceding their 
enrollment in first grade. First grade data are also generally recognized as less 
reliable. 

^°Sistricts have th^^^ students each fall and each spring, and 

thereby measuring school-year gains, or of^measuring studentj only once a year— each 
spring fin each fall — and thereby measuring gains over the entire 12-naonth year. The 
two methods yield remarkably different results, which are discussed in Chapter 4. The 
analysis in this chapter uses only the annual test schedule. 

^^These data include the full range of Chapter 1 students -- successful Chapter 1 
students^ who "graduate" and did hot return to the program the next year, as well as 
the least successful, who may have returned to the program for several more years, or 
who may have eventually been transferred to special education. 
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Ghahges iii Percentile Rente* tor Chapter I Students** 
In Reading and Mathematics, 1983-84 
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Rgure reads: From spring 19M to spring 19W,^^ rihS of second grade students who received 

^^^^^I'^f^^S inmiaion mc^^ to the 31st, while the rank of 12th 
Sracte students remained constant at the I6th percentile. 

Mi^S?^ ^S^^ ^ were MteJiated by first determining ail averages in normal curve equivalents 

""^^ then inverting thwe average to percentile rante. See 
Appendix E for definitions of the measures and Appendix F for the comparable figure printed in NCfe. 

•The number of students included in these analyses varied by grade level and subject; see Appendix G for details; 



Source: M. A. eaipenter and R A: Hopper. Synlhm i^Shaplv I Dabt- Sumn^ Ftoston, VA: Advanced TixSinoiogy. 1985 
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Figure 3.2 suggests that students entering Chapter 1 reading programs tended to 
score at lower percentile ranks than did students entering Chapter 1 mathematics 
programs. The starting percentile ranks for average reading achievement riange from 
16 to 29, while the starting levels in mathematics range from 22 to 35. It also shows 
that students tended to enter both reading and mathematics programs with relatively 
highi r scores in elementary grades and relatively lower scores in secondary grades. 
Students entering second-grade reading programs had average scores at the 29th 
percentile rank, while those entering later elementary reading programs averaged at the 
23rd or 24th percentile rank, and those entering secondary reading programs averaged 
at the 16th to 18th "percentile rank. A similar pattern, though moderated somewhat, 
appears amoiig students participating in Chapter 1 mathematics programs. 

With respect to their achievement at the end of the school year, nearly all (21 
out of 22) of the changes shown in Figure 3.2 indicate an upward movement in 
percentile ranks of students' average scores. The size of these increases is often only 
a few percentile ranks, and Chapter 1 students' achievement at the end of the school 
year was still far from the median, or SOth percentile rank. In general, students 
receiving Chapter 1 services in mathematics demonstrated slightly larger gains than 
those of students receiving Chapter 1 reading services. Their improvements ranged 
from one to seven percentile ranks, whereas reading students increased from zero to 
five percentile ranks. In both subjects, the size of the average annual gain is smaller 
for older students. Whereas elementary students receiving Chapter 1 services ♦ended to 
improve by four or five percentile ranks in reading and six or seven percentile ranks 
in mathematics, secondary students tended to improve by only one or two percentile 
ranks in either subject. Thus, Chapter 1 students in the later grades started with a 
greater educational disadvantage at the beginning of the school year and gained less 
from their participation in Chapter 1 programs than did elementary school children. 
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K^\M^SSUfnmiQns Underl ying the ttf:RS AmOA^ 

The evidence presented here Is based on nationally standardized achievement tests. 
Gains of Chapter 1 students were not compared to gains of comparable students, but 
instead were assessed relative to the entire distribution of scores. For the evidence to 
be meaningful, analysts make two important assumptions. First, they assume that 
students normally progressing through school without Chapter 1 assistance would 
maintain their same relative rank at the end of the school year as at the beginning. 
This is called the "equi-percentile" assumption (Linn. Dunbar, Harnisch, and Hasting 
1982). Second, analysts assume that students not participating in Chapter 1 who score 
at the same percentile rank as Chapter 1 students are in fact comparable to Chapter 1 
students. 

With respect to the first assumption, available evidence suggests that percentile 
ranks are not particularly stable over time. Students may move up or down, relative 
to their peers, during a school year. We do know that there is a strong relationship 
between student test scores on one occasion and student test scores on other 
occasions, thus suggesting that scores do not change radically Over time." If 
individual ranks were as likely to rise as to fall, the changes would cancel one another 
and would not interfere with oUr interpretation of the data. Later, we will present 
evidence suggesting that lower-achieving students tend to fall farther and farther 
behind their higher-achieving counterparts. 



Itjs also known that students with lower test scores will show larger growth 
than other students, a phenomenon known as "regression to the mean" Some 
researchers argue that, because of regression to the mean, these estimates of student 

ISer^"studenS w^'^'n-rr ^^^^^^ ^^'^--^ districts ^Tselect 

Chapter 1 students wjth a different test than they use as a pretest, a practice that 

should reduce the inflation of growth estimates caused by regression to the mean 

Further, the patterns displayed here indicate that groups that begin with relatively 

lower scores also tend to gain less than their higher-achieving counterparts 



23 

V 34 



ERIC 



With respect to the second assumption— that all stadehis scoring at a given 
percentile rank arc comparable— we expect that students selected to receive Chapter 1 
services arc more disadvantaged than those not selected, regardless of the 
comparability of their test scores. Teachers arc permitted to use their personal 
judgment in selecting students to receive program services; If they exercise their 
judgment in a manner consistent with Chapter 1 requirements, they will select from 
among comparably-scoring students those whom they believe to be most in heed of 
Chapter 1 services; that is, students for whom they believe the test scores 
overestimate actual achievement; if the real achievement of Chapter 1 students is 
lower than their starting scores indicate, then their real gains would be larger than 
these shown hcre;^^ 

These problems in interpretation arise in part because the TIERS procedures do 
not include tests of students comparable to Chapter 1 participants to see how their 
achievement actually changes over the year, but instead rely on the hypothetical 
progress of students who began the school year with the same achievement scores.^^ 
This deficiency was addressed in the Sustaining Effects Study, which tested all students 
in its sample of Title I elementary schools as well as some students in elementary 
schools without Title i, in order to directly measure the performance of non- 
participating students; 



^^This analysis assumcs districts place their Ibwesl-achicvihg students in Title I 
programs. In some States and districts, the lowest-achieving students are placed in 
State or local programs, so that relatively higher-achieving students are placed in 
Chapter 1, Even in these districts, however, students selected for service from among 
the remaining unserved children should still be the lowest-achieving students available. 



^;*The TIERS actuaily included several evaluation models which districts could use, 
one of which included a local cbmparisbn group. However, it has only rarely been used 
by districts. The vast majority of districts prefer the model which relies on the norm 
group for comparison. 
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The Sustalftiflg^ Bff eets Stu^ 

the iSustaining Effects Study examined Title I programs in grades one through six 
in a representative sample of schools. The researchers measured student achievement 
before and after one school year (1976-77) in 243 schools and then followed a subset 
of students^^ over two additional school years (1977-79). 

Using the same format as that employed to display the TIERS data. Figure 3.3 
translates Sustaining Effects Study findings into the percentile ranks of the average 
Title I students in 1976-77. Unlike the earlier displays of TIERS data, this figure 
documents students' average entering achievement in the fall of each school year, 
rather than in the spring of the preceding school year, "rom Figure 3.3 we see that 
the average second-grade student entered Title I reading programs in the fall with a 
percentile rank of 26 and finished in the sjsring with a rank of 26; comparable 
percentile ranks for participants in title I mathematics programs were 28 and 29. 

Although the Sustaining Effects Study data were collected eight years earlier than 
the TIERS data, they suggest that students served under Title I were comparable in 
their starting} achievement levels to those served more recently, but that the program 
benefits were somewhat less. Whereas 1983-84 Chapter 1 students improved their 
average reading score by as much as three to five percentile ranks in elementary 
grades, comparable 1976 student averages improved by less than two percentile ranks 
per grade level. Aside from this important difference in the average improvements. 
Figure 3.3 shows patterns of findings that are remarkably similar to those shown in 
Figure 3.2. Both figures Indicate that students entered reading programs at slightly 
lower percentile ranks than they entered mathematics programs with, and that students 
served in earlier grades tended to enter at higher percentile ranks than those served 

"The original sample, on which our findings are based, included 120,000 students 
across six grade levels. 
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Changes in Peit^ntili Railia fef Title I Students 
in Rea ding and iMathematieSi The Sustaining Effects Study, 1976-77* 
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Figure reads: From the fall to the spring testing, 3rd grade students enrolled in Title I reading moved from the 
23rd percentile rank to the 25th percentile. 

*Percent|le ranks presented are based on scores fromLa fell-spring testing cycle in contrast With the spring-spring 
cycle used for TIERS data in Figure 3.2. Changes in percentile rante w^re calculated^ fjrst determining 
anayerages in a starldardized scale sro metric, and then converting these averages ^o percentile ranks . See 
Appendix E for definitions of the measures and Appendix F for the comparable figure presented in NCEs. The 
number of students included in these analyses varied by grade level and subject; see Appendix G for details. 



$bufTO: M. Wana M. Bear, .i Conklin. R;_Hoepfnei; Report 10: Comp&isatory Services and Biucational Dei/elopment In the School Vter 
Santa Monica, CA: System Development Corp., 1981. 
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in later grades. Further, studeixts participating in mathematics programs tended to 
enjoy slightly more relative improvement than did students participating in reading 
programs. 

Because the Sustaining Effects Study researchers collescted achievement data on all 
students in their sampled schools, they were able to compare the achievement gains of 
Title I children to those of comparable students who did not receive services. In 
Figure 3.4, ihe solid lines indicate the percentile ranks of students who received 
Title I services, and the dotted lines show the percentile ranks of students identified 
by teachers as needing Title I services but who were not enrolled in Title I schools 
and therefore could not receive Title I services. 

Figure 3.4 shows that the percentile ranks of needy students in non-compensatory 
education schools often declined. For instance, while first-grade students receiving 
Title I reading services in 1976 maintained their average percentile rank of 30, similar 
students in schools not offering Title I services declined in percentile rank, dropping 
from 29 in the fall to 25 in the spring. These data suggest that improvements in 
Chapter 1 students' percentile ranks may actually underestimate the benefit Of program 
participation. For instance, the n£i benefit of Title I participation for first-grade 
students was thus actually four percentile ranks, rather than zero. 

In nearly all grade levesls, the needy students in non-compensatOry education 
schools lost ground in mathematics during the year, whereas the students who received 
Title I services improved. The patterns were not so clear-cut in reading, however. 
Comparison students in two grades (four and six) rose in percentile rank over the year, 
though in each case they started at higher ranks than did Title I students and did not 
improve as much as the Title I students. Of the 12 comparisons in Figure 3.4 (six 
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Figure 3A 



Changes in PiBr^ritile Rante for Title j SUidents ancLSlmiiar. Students Not Reeelvlhg 
Gompensatory Education, The Sustaining Effects Stucly, 1376-77* 
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Figure reads: From the fail to the spring testing, ist grade Title I students receiving reading instruction 
rhaintaihed position iit the 30th pert^entile rar^fc, while needy students In nonTrtle I schools 
dropped from the 29th percentile rank to the 25th. 

*^rceiltlli rMt« presented aiB ba from a fall^pririg testing cycle in rontrast with the spring-spring 

cj^le used for TTERs d^^^^ by first determining all 

averages In a standardizad scale score metric^ and then converting these av^rag^ b pe r^ntjle rani@. See 
Appendix E for definitions^f the measures and Appendix F for the oompmable figure presented in jsjGEs. The 
number of students included in these analyses varied by grade level and subject; see Appendix G for details. 



SMipe: M. .Wang. J. j^Mri«_R, Hoepfrier. Report 10: Compahsatory Sorvfo&s and Bdacationa! Day^opment in tha Schoot Mtee 
Santa Monica. CA: S/stem Development Corp.. 1981; ^ ^ 
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grade levels in two subject areas), nine are considered statistically significant.^^ That 
is, these differences in growth are likely to reflect something more than random 
fluctuations in test scores. Instead, they reflect real changes. 

The Sustaining Effects Study also examined the achievement progress of Title I 
and needy students in non-compensatory education schools relative to a nationwide 
representative sample of students. Table 3.1 compares the growth rates of these three 
groups. Because these comparisons include students who arc not comparable to Chapter 
1 students, we represent these comparisons in standard deviation units rather than in 
percentile ranks. Changes in percentile ranks are difficult to compare when students 
have widely differing achievement levels. At some places in a distribution of scores, a 
small change in achievement can lead to a large chanjgc in percentile rank. Yet in 
other places, a large change in achievement results ohJy ;n a small change in percentile 
rank. This occurs because the largest percentage of students have scores which are 
concentrated near the average score, while only small proportions of students receive 
very high or low scores. (See Appendix E for a discussion of these measures.) 

Standard deviation units are useful for measuring group differences because the 
size of a standard deviation unit does not depend on where the students' scores are in 
the distribution of scores. Stand.-^rd deviation units are measures of variation among 
students. These measures arc relatively large: the difference between Chapter 1 
students and average students is often just slightly more than a single standard 
deviation unit. If Chapter 1 students begin a school year one standard deviation below 
average and if the average student gains one standard deviation during the year. 
Chapter 1 students would need to gain two standard deviations-twice as much as the 
rcjgular achiever— in order to catch up. 



*^he three comp^rjsons not considered statistically significant arc those for 
reading achievement in grades four, five, and six. 
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Table 3.1 shows that all students gain more in earlier grades than they do in 
later grades. The representative sample gained more than twice as mach in first grade 
as it did in second grade, relative to the variation among students in each grade (1.98 
compared to .87 standard deviation anits). Gains among representative sixth graders 
are about a sixth of their first grade gains (.37 to 1.98 standard deviation units). The 
differences across grades are not so great in mathematics, but the pattern still is 
strong. Table 3.1 also shows that needy students who receive no compensatory 
education gained less than the representative sample in both subjects and in virtually 
every grade level. 

These two groups-the representative students and the needy students-provide 
two very different norms against which to compare the progress of Title I students. 
The evidence presented earlier in this chapter suggested thai Title I/Ghaptcr 1 students 
improved their percentile ranks, or relative standing, but that they still performed far 
below the avcfa^ic student. Table 3.1 shows that these students gained more than 
comparable ^cdy students on nearly every occasion, but still gained less than the 
representative sample on half of the comparisons. 

Table 3.2 further pursues the differences between Title I gains and the gains of 
these other two groups by representing the gains of Title I students as a percentage of 
the gains of these other two groups. It shows, for Instance, that Title I students in 
first-grade reading gained 90 percent of what the representative sample gained, but 
that they gained 119 percent of what needy students gained when they received no 
conipcnsatory education. That is. Title I students gained 19 percent more than these 
other needy students who received ho services; but 10 percent less than the 
representative sample of students. In mathematics the gains of Title I students range 
from a 10 percent increase in third grade learning rate to a 31 percent increase in 
sixth grade learning rate, over the learning rates of comparable needy students. Yet 
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Table 3.1 



Growth of three Grbug^s of Students Participating 
in the Sustaining Effects Stiid^, 1976-77 
(Expressed in Standard Deviation Uhits^) 



Representative 
Sample^ 



Title I 
Students^ 



Needy Students 
With No CE^ 



Reading 



Grade 1 


1.98 


1.79 


1.60 


2 


.87 


.85 


.77 


3 


.6i 


.64 


.53 


4 


.46 


.50 


.49 


S 


.42 


.38 


.34 


6 


.37 


.37 


.37 


jyiath 








Grade 1 


1.75 


1.76 


1.40 


2 


1.24 


1.19 


1.04 


3 


1.21 


1.13 


1.03 


4 


.84 


.90 


.79 


5 


.70 


.68 


.55 


6 


-58 


.64 


.49 



i/ All gains are converted to standard deviation units, using the standard deviation 
of the Fall scores of the Representative Samples. 

2/ Data on representative students :akcn from Tables 1-2 and 1-3, pages 9 and 10 of 
Report JO. 

3/ Data on Tit^s I students from Table 2-2, page 40, Report 10. 

4/ Data on comparable needy students from Table 2-5, page 43, Report 10. 

Source: M. Wang, M. Bear, J. Cdnkliri, R. Hbepfner, Report 10: eompensatorv 

Services and Educational b&ve4A 0ment frh4He-SchoQl Year: Santa Monica, 
eA: System Development Corp., 1981. 
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table 12 



Gains of Title I Students as a Proportion 
of Gains of Other Students 
(From Table 3.1) 



Title I Student Gains as a Proportion of ^ 

Rcprcsehtative _ Needy: 

Sample Gains Student Gains 



Grade 1 90% 119% 

2 98 110 

3 105 121 

4 109 102 

5 90 112 

6 100 100 



Kin tfi 

iVlCl ill 



Grade 1 101% 126% 

2 96 114 

3 93 ilO 

4 107 114 

5 97 126 

6 110 131 



i/ Title I student gains shown in Table 3^1 arc shown here as 
proportions of the gains of other groups. 

Source: M. Wang, M. Bear, J. Cbnklin, R, Hoepfner, Report 10: Compensatory 

Services and Educat ional Development in the School Year. Santa Monica, 
CA: System Development Corp., 1981. 



32 



ERIC 



43 



they stiii gained iess than the representative sample in three of the six grades 
examined. 

It is also possible to present comparisons such as these graphicaliy, though the 
graphic representation requires still another scaling device. Using a method of scoring 
the achievement tests that allows growth across all the grade levels to appear within a 
single scale^ Sustaining Effects Study researchers were able to illustrate the 
achievement progress of three groups^ two of which we have already discussed: Title I 
students and needy students in non-compensatory schools. In this analysis, however, 
the Sustaining Effects Study researchers did not use the representative sample for its 
third group, but instead used students not receiving cbmpehsatbry education though 
enrolled in Title I schools.^^ This contrast serves roughly the same purpose as the 
preceding one, in that this group indicates the progress of students who arc not 
considered to be in need of any special services. Figure 3.5 shows the achievement 
growth experienced by these three groups of students at each grade level. Within each 
group, we have superimposed the progress of students of different ages into a single 
figure, so that it simulates the hypothetical progress of a single group of students 
moving through the entire elementary school sequence. Students labeled in Figure 3.5 
as "Title I students" and as "Needy students ib non-compensatory education schools" 
are the same groups shown in Figure 3.4. The growth patterns look different because 
this new scale is designed to show students' actual learning over time, whereas 
jpigure 3.4 uses a percentile rank scale in order to show changes in relative standing 
over time. Thus, the downward lines in Figure 3.4, showing a loss in percentile rank, 

^^Nqn-compcns Title I schools arc not the same as those in the 

representative sample used in T^^^^ The representative sample included 

students in Title schools wliqjvcre not receiving compensatory services, presumably 
because they were less in need of services, and students in non-Title I schools who 
were not identified as needing compensatory services. Figure 3.5 includes only the 
first of these groups. 
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Figure 35 



Reading and Matjiema^ Achie/ement of Studente Receiving 
and Not Receiving Compensatory Education, Sustaining Effects Studjr* 1976-77* 



\AE)rtical Scale Score* 
650 



600 

550 
500 
450 
400 
350 



Reading 



NDn-Ck)mpertsatory Education students in 
Title I schools 

Title I students 



30^ — 



} L 



students in non-Ck)mpGnsatbry 
Education schools 

I I . I I 



1 



Grade 



\^rtical Scale Score* 

Mathematics 

600 
556 
500 
460 
400 
350 
300J^ 



J L 



J L 



1 



6 



Grade 



Hgure reads: The vertical scale scores of title I first-grade student for reading and mathemati^ iricre^ed 
rnbr© from the fall to the spring than did those of similar students not enrolled in Htle I schools, 
yet Title I first graders started t^hind regular Jirst graders in Htle I schools who did hot receive 
Chapter I and fcdied to catch up by the spring. 

"The achievamerrt treads printed here differ frbrn th^ previousiy used because they are expressed Iri 
verticlU s(^le scored percentile ranks, A group can gain in achievemertt but still show decreases 

in percentile ranks relative to the entire student population. 

^^ygjlicaL scale srorfs are bhe fbrrn of explmded s(^e allow corTiparisoris across grade and 

cx)ntent levels. See Appendix E. The number of students Included in these analyses \^ed by grade level and 
subject; see Appendix G fbr details. 



Sbiirce: M. Wang, M. Bear, J. Cbrildrn, R. Hbepfhen Report 10: Ck>mpensato/y Servkeis end Bducethnei Da/ehpment in the Schoot Vtor 
Santa Monto, CA: SysEtem Developmem Corp., 1981. 
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do hot iheah that students actually forgot material^ but rather that they learned less 
than other students. Figure 3.5 shows the amount these students actually gained. 

Figure 3.5 demonstrates several important trends. Firist, Title I services reduced 
the gap between Title I students' achievement and the achievement of more advantaged 
non-compensatory education students in Title I schools^ but only slightly. Second, 
bearing in mind that these arc not the same students followed over six years, but 
rather six groups of students, each followed during the 1976-77 school year, the data 
nevertheless indicate that the gap between disadvantaged students and regular students 
is much larger in iater grades than in earlier grades. Although Title I students 
generally gained more during individual years than comparable needy students in non- 
compensatory education schools, their rate of growth was not enough during any one 
year alone to bring them substantially closer to the performance of the more 
advantaged hbh-coixipehsatbry education students in Title I schools. 



Effeets^Studv Analys e s 

The Sustaining Effects Study researchers assessed the progress of Title 1 students 
by comparing it both with the progress of comparable "needy" students who received 
ho services and with the progress of nbn-cbmparable students who did not need 
services. The validity of these comparisons depends oh two assumptions. First, these 
comparisons assume that the students in non-Title I schools identified as "needy" were 
indeed "cdmparable" to Title I students. Yet we know they attended schools which 
served relatively fewer poor students thah bthcr schools in their districts, and our first 
report to Congress showed that students attendihg these schbols generally achieved 
more thaii students attehding schbbls servihg higher cohcehtratiohs of poor children. 
Thus the performance bf these needy students, even thbugh it often went down relative 
to national norms, was probably better than we would expect of Chapter 1 students if 
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they Had not received program services.^® 

Second, these comparisons assume that students labeled as **needy** who did hot 
receive Title I services also did not receive any other special educational assistance. 
If these low-achieving students received extra instructional services jbecause of their 
low achievement— and there is evidence suggesting that they did (Garter, 1980)— then 
their achievement gains would reflect that assistance and would overestimate the size 
of the gain we could expect in the complete absence of compensatory education 
services. Hence, the benefit of Chapter 1 assistance may be underestimated in our 
analysis if the comparable "needy students in noncbmpehsatbry schools'* in Figures 3.4 
and 3.5 actually received other forms of educational assistance. 

Statistical Corrections for Noncomparable Groups 

In recent years new statistical estimation techniques have been developed that 
more effectively correct for group differences arising from the program's selection 
procedures. As part of the National Assessment of Chapter 1, we applied these 
statistical techniques to the Sustaining Effects Study data (Myers, 1986). In this 
analysis^ we formed several different cbmparisbh grbups whose achievement growth 
could be used to estimate the expected growth bf Title I participants if they had not 
received Title I services. One of these consisted of all nbn-participating students in 
Chapter 1 schools, and another included all nbn-participating students in non-Chapter 1 
schools. The third consisted of a statistically-created group which was comparable to 
Chapter 1 students ^oth in entering test scbres and in family background 
characteristics. Reanalyses of Sustaining Effects Study data using these analytic 

^®The avcfage sbcioecbnbm bf students in a schbol has been shown to be 

strongly related to individual student's achievement. Our first interim report to 
Congress summarized sbme of this evidence. Other examples of these findings can be 
found in Coleman^ Campbell, Hbbsbn^ McPartland, Mbbd, Weinf eld, and York (1966, 
known as the Equal Fdiicational Opportunity Study-EEOS), McPartland and York 
(1967), Wolf (1977) and White (1983). 
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techniques indicated that the more similar the comparison group was to Title I 
participants, the greater the achievement benefits associated with Title I pafticipatidn. 
However, the patterns of effects found when using the statistically optimum comparison 
group was not substantially different from that found by the original Sustaining Effects 
Study analysis; 

Variability of Outcomes 

The findings from these studies should not be taken to mean that effects shown 
here would be the same for all districts, schools, or students. Many students make 
larger achievement gains, while others lose ground. Even when scores are averaged 
across school districts or over entire States, we still see variations in average gains. 
The State reports submitted to the Department of Education under TIERS illustrate this 
variability. Table 3.3 presents the range in the average gains reported by 
States.*® It shows, for example, that State averages in second-grade reading scores 
ranged from a loss of 5.7 points to a gain of 7.9 points.^^ Large differences among 
State average gains are evident in other grade levels as well. Ninth-grade mathematics 
gains ranged from -2.1 to +15.7 points across States. Further, these variations among 
States are roughly consistent from grade to grade. That is. States which report small 
or negative gains in one grade are likely to report similar results in other grades, 
whereas States which report large gains in one grade are also likely to report large 
gains in other grades. 



^^Table 3.3 includes only those States that reported 500 or more students for a 
particular grade level. Small sample sizes ten^^o be affected by extremely low or 
high scores. By excluding States with small numbers of students in a grade, the 
results shown in Table 3.3 are more stable and less likely to have been affected by 
extreme student scores. 

^°Table 3.3 reports achievement gains in the statistical metric used by TIERS, the 
normal curve equivalent (NGE). Its properties are discussed in Appendix E. 
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table 33 



Range |h State-Repbi^ J^erage Gaiiis* of 
Chapter 1 Students in Reading and Mathematics in Grades 2 Through 12, 1983-84 









iMathematics 






Range Of 




Range of 


Grade 


Ay/erBge gain 


State averages 


A^^rage gain 


State ^rages 


2 


1.0 


-5.7 to 79 


3.2 


-1.2 to 1l5 


3 


ad 


-05 to ao 


32 


-25 to 11.4 


4 


29 


-05 to 9.6 


31 


-2.1 to 7.6 


5 


ai 


-05 to as 


4.4 


-30 to 9.0 


6 


3.2 


-03 to 69 


4.0 


-2.6 to 67 


7 
f 




—1 A in 

— 1 .H lU O.D 


35 


-1.0 to a4 


8 


2.4 


-1.6 t o 53 


31 


-1.1 to 7.6 


9 


1.6 


-04 to 95 


0.7 


-2.1 to 157 


10 


1.1 


-1.6 to 6.7 


05 


-1.9 to 2.7 


11 


03 


-4.7 to 1.7 


1.1 


-2.8 to 35 


12 


03 


-55 to 23 


1.9 


0.6 to 43 



l&ble reads: From spring 1983 to spring_19S4» second grade Chapter I students on avemge gained 1.0 NCE in 
their readihjg scon^ NCEs in their mathernatjcs However, State gsuhs average 

reading scares ranged from -&7 to 75 NCEs while in mathematics they ranged from -1.2 to IIS 
NCEs. 



nrhe_ average gains presented combine scores for all states submitting rejx)rts. Ranges include only those states 
rei)ortirig scores on 500 or more students. Gains are expressed as NCE scores, a metric explained in 
.Appendix E. 



Source: M. A. Carpenter and P. A. Hopper, Synthesis of Chapter I Data: Summary Report Re^n, VA: Advanced lechnotogy, 1985i 
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There are a number of reasons why such differences exist Program effects may 
vary due to differences in the grade levels or subject matters emphasized, the 
characteristics and heeds of students participatinjg, the extent to which students reside 
in areas of high concentrations of poverty, the degree to which the achievement test 
matches the curriculum, the nature and extent of States* oversight of local Chapter 1 
programs, the quality of the local Chajpter 1 programs^ the quality of the regular 
programs, or the way in which the two programs are coordinated. Many States also 
sponsor compensatory education programs and these may interact differentially with 
Chapter 1. 

Program Ef fects^n Stxideiit^titUdes Toward^cho^ 

A decade ago, there was considerable interest in student attitudes toward school. 
Several analysts suggested that attitudes were more important to student achievement 
in the long run than were short-term achievement gains. This view suggested that a 
program designed to alter such attitudes could have greater effects oh ultimate 
educational achievement than a program concentrating directly on basic skills. Two 
studies of Title I-the Instructional Dimensions Study (NIE^ 1976; Coblcy, 1978) and the 
Sustaining Effects Study--addressed that hypothesis by measuring student attitudes 
toward school in addition to student achievement gains. The Instructional Dimensions 
Study's analyses of students* attitudes towards school revealed ho significant changes 
from jfall to spring in either grades one or three^ the only two grade levels included in 
the study. However, the students exhibited very high scores on the attitude 
instruments in the fall, so that there was little room left for scores to increase by the 
spring. 

Researchers involved in the Sustaining Effects Study found inconsistent patterns 
in attitudihal changes of participating and nonparticipatihg students across the six 
grade levels and two subjects of Chapter 1 instruction. These differences were further 
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compiicated by overall changes in attitude. For instance^ ali students improved their 
attitudes in second grade, yet all became more negative during sixth grade. 

Some observers have proposed that other indicators, such as attendance records or 
disciplinary records, be used instead of paper and pencil instruments to measure 
student attitudes. To date, no national data include such measures. 

Tl^^^tetionshib^etwcKsn Cbst^^^^^ 1 Services 

In addition to studying the effect of program services on students* standardized 
achievement test scores, many investigators have also tried to determine the 
relationship between program effects and the amount of money spent to achieve those 
effects. But research on this relationship, between program costs on one hand and 
program effects on the other, has proved to be difficult to conduct for several reasons. 

First, estimates of effects by themselves have been difficult to determine, even 
apart from their relationship to costs. We have already seen that the size of ah effect 
may vary across grade levels, subject matters and ty|5es of students, and that the 
researcher's estimate of the size of the effect depends heavily on the choice of a 
comparison group against which to compare prbgranl beneficiaries. 

Second, cost-benefit studies are designed to measure the influence of costs on 
achievement. Yet decisions about costs--that is, about how to allocate resources— may 
also be influenced by achievement. Suppose a district allocates its most expensive 
resources to those students who have the most difficulty learning, while allocating its 
cheaper resources to eligible students who have less difficulty. And suppose, as a 
result, all its students gain comparable amounts during the school year. Unless a 
researcher were able to take these allocation decisions into account, a feat which is 
difficult to accomplish, the data gathered in this district could indicate no relationship 
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between costs and effects. For it would show that siudetit growth was uniform, while 
costs varied considerably. 

Finally, districts differ substantially in the nature of their student populations, in 
their decisions about how to allocate Chapter 1 funds, and in the level of State and 
local funding available to serve these students. Even though districts may vary by 
several hundred dollars in their Chapter 1 pcr-pupil expenditures, they vary by several 
thousand dollars in their total per-pupil expenditures. In the context of these vast 
differences in total expenditures. It becomes extremely difficult to isolate the 
relationship of Chapter 1 expenditures to Chapter 1 effects. 

Despite our reservations about research oh this topic, some researchers have 
taken the available evidence as conclusive. Mullins and Summers (1982), for instance 
review the available evidence and conclude that there is no relationship between 
compensatory education program costs and benefits. Indeed, a number of investigations 
have been undertaken to determine the relationship between costs and benefits, and 
most have been unable to document such a relationship. However, because of the 
difficult methodological issues involved, our conclusion from reviewing this literature is 
that researchers have not yet developed adequate methods for determining the 
relationship between program costs and program effects on standardized achievement 
test scores. 

Summary and Discussion 

The analyses described in this chapter lead us to these conclusions: 

2- Students receiving Chat>te^^^-sej ^yices exb e ncn ^^ increases jn theiV 

Standardized achievcitlent test sco res than c omparable students who do np t 
However, their gains^o not m o ve^hem substantially toward the achieve ment 
levels of more ad vantaged st^ldents. 

The two data sets reviewed here, while differing substantially in date of data 

collection (1976 versus 1983), identity of data collector (independent researchers versus 
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school districts), use of comparison groups, and sizes of average gains demonstrated, 
provide us with remarkabiy similar patterns of achievement tiest scores. They both 
indicate that students participating in local Chapter 1 programs increase their 
achievement rank, jelativc to comparable students who received no services. Both also 
suggest that these gains are not enough to close the gap between Chapter 1 students 
and their more advantaged counterparts. The two data sets also indicate that 
Chapter 1 benefits vary from almost none in some subjects and grade levels to 
relatively substantiai benefits in others. Because of the way in which Chapter 1 
services are allbcated, with students who gain the most removed from the program 
each year and those who benefit least retained. It is not possible to know whether 
benefits such as those shown here would add up across the years to yield an aggregate 
multi-year benefit. 

An important difference between the Sustained Effects Study and the TIERS 
data had to do with the sizes of the average improvements. Whereas the Sustaining 
Effects Study, conducted during the 1976-77 school yeav, found elementary reading 
averages to improve no more than two percentile ranks during the school year, TIERS 
data collected during the 1983-84 school year show elementary reading score averages 
improving by two to five percentile ranks, depending on the grade level. And whereas 
the Sustaining Effects Study showed average mathematics scores generally improving by 
three or four percentile ranks during a school year, TIERS indicates that such scores 
improved by five to seven percentile ranks. These differences are particularly 
remarkable in light of the fact that the Sustaining Effects Study used a fall-tb-spring 
testing schedule, a schedule now generally believed to Inflate growth estimates 
(Keesling, 1981); 

Several hypotheses could be generated to account for these differences, but two 
are particularly important. One is that program administrators and teachers have 

42 



53 



actuaiiy improved their practices in the eight years between these two analyses^ and 
that the TIERS data reflect the clffects of these improvements. The second hypothesis 
is that, because tllERS permits local districts to choose their own tests* whereas the 
Sustaining Effects Study used a common test across all districts and schools* the TIERS 
data more accurately reflect the instructional objectives of local Chapter 1 projects 
and consequently are able to document program effects that a common test might not 
measure. Freeman and others (1983) have shown that the extent to which test content 
matches the content of textbooks can vary substantially from one combination of 
textbook and test to another. Each of these hypotheses probably has some merit, but 
the relative importance of each-or of ethers-cannot be ascertained from the available 
data. 

3. Students barticipating in Chapt er 1 mathematics programs 

gain more^tfaan those narticipating in Ch anter 1 reading propfam?; 

Findings from both TIERS and the Sustaining Effects Study indicate that 
mathematics programs yield larger gains in student achievement than do reading 
programs, this was true for grades two through twelve in the TIERS data and for 
grades one through six in the Sustaining Effects Study. When the Sustaining Effects 
Study researchers applied statistical tests to their data, they found that reading 
programs yielded statisticaiiy significant gains only in grades one* two* and three 
whereas mathematics programs yielded significant gains in all six grades. 

Several hypotheses have been put forward to account for this finding. One 
hypothesis is that mathematics is provided to students who have higher entering 
achievement scores than students in reading programs and that such students arc more 
likely to benefit from compensatory education services. Another is that reading and 
mathematics differ in the extent to which they are taught putsi^g of Chapter 1. 
Because reading is a task required by many school subjects, the extra instruction 
provided by Chapter 1 niay constitute a smaller portion of the student's total reading 
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practice during a school day; N&thematics, bh the other hand, is not used in as many 
other school subjects and thus the additional mathematics instruction provided by 
Chapter 1 prbgraihs may constitute a significant addition to the students* total 
exposure to mathematics. 

4. St u^ems^4n-ear lv cl e ih c htarv Chapter 1 programs gai ri inore ^^mtt^^Hleftts 
partieipa ^irig in later - g rad^^K^Tams. 

The TIERS data show a consistent pattern of decreasing gains as students moved 
from elementary to middle to secondary grades, with program effects almost negligible 
in the secondary grades. The earlier Sustaining Effects Study yielded a more mixed 
pattern within the elementary grades. 

Why such differences occur is not clear. Gains in upper grades could be smaller 
because students start further behind in these grades, and generally have more 
difficulty learning; because they are more disaffected with school and are less 
motivated to learn; because the academic content taught in these grades is more 
difficult to learn; because the services provided are hot as good as those provided to 
younger studen«:s; or because sccondiary-level achievement tests do not cover the types 
of basic skills taught by secondary Chapter 1 programs. 

5. Evidence regarding program e ffects on student attitudes toward school is 
inconciusive 

Though two studies of Title i attempted to measure changes in student attitude 
toward school, and to attribute these changes to Chapter 1, neither was successful. 
Part of the difficulty is in measuring attitudes by means of paper-and-pehcil 
instruments. Some analysts now advocate the use of such indicators as attendance 
rates or participation in extracurricular activities to estimate attitudes toward school. 
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6- Researeherg Save not vet deveioo ed adequate methiods^fSf-^t, , ... 

rg l atjpnshiP between nrogram costs and orog r am effects sn^tanaaTdized 
a^jevement^est scores: 

Though a few studies have attempted to relate program costs to program benefitSi 
the task has proved to be extremely complicated and no studies to date have been able 
to solve the many methodological problems presented; 
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4. Siistatning Achievement through the Summer and Future School Years 
Central Findings 

7. The achievement gap between disadvantaged arid advantaged students appears 
to widen during the summer months. 

8. Tit^e I-supported summer programs have not narrowed the achievement gap; 
however, these programs often were not designed to be academically 
rigorous. 

9. Students who discontinue Title I appear gradually to lose ground when 
they no longer receive services. 

10. Chapter 1 students with very low achievement levels appear to maintain their 
relati^ve achievement levels while participating in Chapter 1, but not to move 
ahead. However, the evidence suggests they would have lost ground relative 
to their peers if they had not received services. 

11. No nationally-representative the long-term effect of 
Chapter 1 programs on graduation rates, future education, or adult literacy. 

Patterns of achievement test scores among disadvantaged students suggest that 

these students not only score lower, on average, than their more advantaged peers, but 

that their scores fall farther and farther behind their more advantaged peers as they 

move through school. Why this happens is not understood and has been the subject of 

both speculation and research, some of which will be described in this chapter. One 

hypothesis proposed by Hayes and Grether (1969) received considerable attention in the 

1970s: disadvantaged students learn as much as other students during the school year, 

but they forget more or learn at a lower rate during the summer. Such a phenomenon 

would be consistent with one of the underlying premises of the Federal compensatory 

education program: students intended to receive Chapter 1 services were assumed to 

come from homes that could not prov.iJe the enriching experiences that more affluent 

homes provided. It was assumed, for example, that the homes of disadvantaged 

students would offer fewer reading materials and fewer opportunities to engage in 

learning activities. If this is so, then disadvantaged students ^rould lose ground, 

relative to their more advantaged peers, each summer. 
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The possibility of a summer "dropoff" in the achievement of disadvantaged 
students raised three questions. First, was it true? Second, if it was triie^ would 
summer school programs supported by Chapter 1 prevent such losses? And third, did 
the achievement gains that occurred as a result of Chapter 1 services during the 
regular school year last through the summer^ or were they also lost? Each of these 
questions has received research attention. 

Extent of a Summer Dropoff in Achievement 

With regard to the first question— is there a summer dropoff in the achievement 
of disadvantaged students— two studies of Title I examined students* test scores over 
time: the NiE Compensatory Education Study (NiE, 1978) and the Sustaining Effects 
Study (Klibahoff and Haggart, 1981). Both found that the achievement of 
disadvantaged students' did in fact decrease during the summer. Instead, it 
increased. However^ it did hot increase as much as did the achievement of advantaged 
students. Thus, the notion of summer loss did not account for the increasing gap in 
achievement as students progressed through school, but differences among childrens' 
rates of increase over the summer did. 

Table 4.1 shows the findings from the NIE Compensatory Education study. The 
table presents both school-year and summer gains in the achievement scores of 
compensatory education students who began the school year below the 50th percentile, 
and noncompensatory education students who began the school year scoring above the 
Sbth percentile. These data indicate that all students gained more during the school 
yeaf^^ than during the summer, but that most groups gained during the slimmer as 
well. It also indicates that summer gains were larger in reading than in mathematics. 

^^The gains of Chapter 1 students shown here are not representative of 
nationwide averages. Because the NIE wanted to study the relationship between 
project characteristics and student outcomes^ it purposely selected well implemented^ 
stable projects for its study. 
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Table 4.1 



Gains in Achievement fer eompehsatoiy^^ Education Students* 

During ttie School Ybar and Summer, 1976-77 



Achievement Gains 
[Expanded scaie scores]' 



Dbmpehsatory 

Education 

Students 



Non^mpensatoiy 

Eddcatioh 

Student 



School year 
gain 



Summer 
gain 



Reading 


Grade 1 69 b 

j3rade 3 44 5 




Mathsmatios 


Grade 1 43 2 
Grade 3 64 -1 


Reading 


Grade 1 55 -jo 
Grade 3 36 21 


Mathematics 


Gmde 1 39 6 
Grade 3 7 



T&ble reads: Compensatory education students who were in 1st grade and receivLng reading services gained 69 
uriite In readin:t oyer the school year but 0 units over the summer In comparisqri, 1st grade pupils 
nctf receiving compensatory reading instruction gained 56 units in their reading score during the 
schcx)! year ani 10 units during the summer. 



•Studente in oompensatpry educatidh ihcfuded only those whose pretest scores were below the 50th peft»ntile- 
those not receiving compensatory education Included only those whose pretest scores were at or above the SOth 
percent i le* 

**Gains are expressed as a forrn of expanded standard scores; these are defined in Appendix E; 



SgSroC. 'SmSnaJf '^^'"P^'*^ sm Final to Cong^ From the National InstltutB of BiucaUon. 



49 



59 



ERIC 



Most important for Chapter U however^ is the difference mmer gains between the 
compensatory education students and other students. Iii both subjects and at both 
grade levels, noncompensatory education students gained rr.ore during the summer than 
did compensatory education students. 

These patterns were corroborated by Heyns (1978) in one of the most influential 
studies to date on this issue. Heyns analyzed data from one large urban school system 
and concluded that schooling mitigates the increasing achievement gap among students 
from diverse backgrounds, and that the summer months increase the achievement gap. 
Table 4.2 presents a portion of Heyns' data. Heyns' datn do not identify Chapter 1 
students specifically, but they compare the sumnler gains of low income students and 
higher-income students. Students from families with low incomes displayed 
substantially smaller summer gains than did students from families with higher incomes. 
This difference is especially pronounced among black students: black sixth graders 
from low-income families demonstrated achievement score losses over the summer in 
word knowledge, while black students from higher-income families showed gains. To 
the extent that these summer losses among low-income blacks, and smaller-than-average 
gains among low-income white students, are cumulative they help explain the growing 
achievement gap between disadvantaged and other students. 

in a more recent paper, Heyns (19^6) used the Sustainin Effects data to 
illustrate the widening gap between low and high-achieving students. Figures 4.1 and 
4.2 show the achievement growth patterns for students in the lowest, second, and third 
quartil^s. In each figure, the school year and summer growth of students at all grade 
levels are superimposed to simulate the growth of a single group across six grades. 
The scores are presented in vertical scale scores, a metric that permits student 
achievement in all six grade levels to appear on a single scale. Both charts suggest 
that groups who enter first grade with different achievement levels can be expected to 
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lable 4.2 



School YlNar aiicl Summer Achieverfieht Sains ^ Race and 

Family income for sixth 6rade Pupils In One targe Urban District, 1971^ 
(Gitide Equivalent Scores)* 



Race and income 


rail 

19^ 


Spring 
1972 


call 

rail 
1972 


School 
gains 


Summer 
gain 


N 


Nationaj average** 


5.10 


5.80 


&10 




3d 




Ibtal samble*** 


4 9*\ 


HJOi 


4.00 


.62 


-.01 


1,445 


White 


4m 


550 


&b4 


£4 


.24 


459 


bass than $9,06d 


4.21 


456 


4.93 


.65 


07 


IBS 


$9.000 -$14^ 


4.77 


5.73 


531 


^96 


.18 


115 


$15^ + 


5S6 


086 


7.15 


1.00 


.29 


124 


Black 


333 


4.44 


4:32 


si 


-.12 


986 


bess than $4,^00 


a62 


4.04 


aTB 


.42 


-.28 


187 


$4,000- $8^ 


354 


435 


4.23 


SI 


-.12 


325 


$9flQ6 - $14^ 


4.08 


4.67 


455 


59 


-.12 


169 


$15fl00 + 


457 


5.19 


54l 


.62 


.22 


101 



ISble reads: From fall 1971 to fall 1972, the perfonnanc© of eth grade white studentsjn the sampib . creased £4 
grade equivalent units during the school year and .24 unite during the summer. 



**:hieverTient Is i^pressod^ studente' mean grade equiv^ents»res on the Wsrd Knowledge subtest of the 
Metropolitan Achievement Test,- grade equivalent soores are described in Appendix E. 

**These are the mean grade equivalent scores predicted fer the particular test dates. 

***The sample consists of all students with test scares as^lable at all three dates. Totals include students for 
whom family income was missing. 



Sounce: & Heyns. Summer Learning and the Bfects of SchooSng, New Vbrk: Academic Press. 197^ pp. 45, 282-^ 
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Rgure 4.1 



>^rticai^cale Sieores for Reading as a Fuhctibn 
of Grade t^l fojLSd Percentiie Ranks* 



vertical Scale Scores** 
700 r— 




Figure reads: A beginning first grade student scoring at the 25th percentile, vvhich corresponds lo a vertical 
scale score (VSS) of 326, typically received a VSS the next spring of 375 and a VSS of 386 the 
following fall as a second grader. 

*The zig-zai nmure oM "sum rrier drop-off". The "negative 

growth'' observed for some spring to fall intervals may be due to sample and test-level differences. 

**Vertical Scale Scores are expanded scale scores as defined in Appendix E. 



Source: J. Hemenws^r M. Wang. C. Kanoyer; R. Hoepfner; M. Bear, and G. Smith. Fiepon 9; The Measures and \/kriabfes In the Sustaining 
Bfects Stuefy Santa Monica. CA: System Development Corp., 1981. 
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Figure 4.2 



Vertical Scale Sk»res for Mathsmatics as a Fuhetleh 
of Grade bevel for Selected Percentile Rante* 



vertical Scale Score" 
7D0|— 




400 — 



350 — 



300 



75th . 
Percentile 



50th 

Percentile 



25th 

Percentile 



IB 2.1 



2.8 ai 38 4.1 AS 5.1 

Grade/Test Administration 



53 &1 



6S 



Figure reads: A beginning first grade student scoring at the 25th percantileL which cdnBspdhds to a vertical 
scale score (VSS) of^3lCt typically received a VSS the next spring of 362 and a VSS of 3^ the 
fblloMring ^1 as a second grader. 

•the zig-z^ nature of the curv^ should n^ be attributed nec^i^'ly to "summer dn}(><rfr'. The "negative 
growth" observed for sorn© spring to fall Intervals rnay be due to sample and test-level differences. 

**V9rtical Saaki Boobs are eo^anded scale scores ea defined In Appendix E. 



S(XiR»^iL JHk9ffiimqr.M VMuig, C Ksnoyei; R. Hoof^hei; M. Bear, and Q. Smith. Beport S: The Measures ena V&^6tos In the Sustaining 
BMa Sftjoy Santa Moties, CA: Sygtom Oewlopnient CorpL, 1981. 



53 



e3 



ERIC 



grow farther apart as they move through the elementary grades. The iater-grade 
difFerences are even greater in reading than they are in mathematics. Because 
Figures 4 A and 4.2 superimpose the achievement growth of separate grouiDS of students, 
one at each grade level, to create the appearance of a single group moving across the 
grades, the summer "growth" is difficult to interpret. Both figures indicate, though, 
that students beginning school in the fall often have scores that are lower than those 
of students completing the preceding grade. And, in reading at least, the summer 
dropoffs are more severe among low-achieving students than among high-achieving 
students. 

These increasing disparities amoiig achievement scores are particularly noteworthy 
when viewed together with the amount students learn each year. During first grade, 
for instance, students at all achievement levels gain a substantial amount. Yet the gap 
is already v idening, and by the end of first grade students in the lowest group have 
scores comparable to those held by the highest group at the beginning of first grade. 
By the time students reach sixth grade, their achievement scores are much more 
diverse. Yet, as we saw earlier, the amount students gain /iri?ug the sixth grade is 
much less, relative to the differences among high- and lovv rchu'ving students, than it 
is in earlier grades, especially in reading. Thus students in v ; lij^i^t g: cup begin the 
sixth grade with reading scores comparable to those held by ^ . v hest jr oup when 
they began third grade, and do hot gain nearly lenough durin^^ t ? jC;xt to cose the 
achievement gap. By the end of sixth grade, their achievement siPlt :::omp£rable to 
the tiiird*grade reading achievement of the highest grotip, 

ImiDaet of Summer School Programs 

The notion that summertime may contribute to the widening achievement gap 
sparked interest in the 1970s in the hypothesis that Title I summer projects might 
either prevent summer losses or promote further growth in the achievement scores of 
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educatibrially-disadvantaged students; Both the iSustainiiig Effects Study researchers 
and Heyns examined the effects of participation in summer programs. 

The Sustaining Effects Study identified students who jjarticipated in Chapter i 
suinirier programs and compared their achievement growth with that of comparable 
students not participating in summer programs. Figure 4.3 shows the resulting 
achievement growth patterns. It suggests two important trends. First, compared with 
students who did not attend summer school, stu * -ts who attended summer school had 
lower achievement scores initially and thus were more in need of compensatory 
education services. Second, summer school programs had very little apparent effect on 
these students' achievement. In most cases, the achievement growth of summer school 
students was still less during the summer than it was during the school year, and it 
was rarely greater than the summer growth of the higher-scoring non-participating 
students. The Sustaining Effects Study analysts attributed this finding to the fact that 
Title I summer programs were typically brief, averaging about six weeks, and that the 
summer instruction often was neither intense nor geared specifically towards increasing 
basic Skills (Klibanoff and Haggart, 1981). Thus, the programs could not realistically 
be expected to affect achievement test scores. Heyns' research also indicated that 
summer programs are generally not rigorcas in an academic sense and thuo cannot be 
expected to raise achievement test scores (Heyns, 197g). 

Retention of Chab teiHHBenefits Qftr med 
During thv> . ^4ib&i^ Yeaf 

The third important question regf^iding t irelationshlr. between summcf and 
school-year learning was this: since th.; acbieveo'fwi ^jap be*v en disadvahtag^a and 
other students seems to widen over tj vvh ■ ;nt do th v>enefits of school year 

Chapter 1 services remain through the ^umra?.. r.: . c > the foilv.ing school year? 
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RguiB 4.3 



Gains Jn Achievsmerit fbr Title L Students Attending 
and Nbt^ttehding Summer School Sumrner 1977 



Vertical Scale Scores* Vertical Scale Scores* 




Fall 76 Spring 77 Fall 77 Fall 76 Spring 77 Fall 77 



Figure reads: Tiye I students who^w^^^ during fall 76 who did not attend surhmer school had slightly 

higher test scores at all three testing periods (fall 76, spring 77, and fall 77) than did students 
who attended sumrner projects. 

''yertical sca^^^^ are a form of expaulded scale scored as defined in Appendix E. The number of students in 
these analyses varied by grade level and subject; see Appendix G for details. 



Solirte: L. Klibanoff and S. Haggart. Report 8: Surhmer Growth arid the Bfectiveness of Sumrner School. 
Santa Monica, CA: System Devetopment Corp., 1961. 
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TIERS provides one source of evidence on this tbjjic. Under TIERS; school 
districts can clioose from several options in deciding when to test students; They may 
test students once each year, every spring or every fall, or they may test students 
twice a year, in the fall and again in the spring. When the former schedule is used, a 
single test serves both as a posttest for one school year and as a pretest for the next 
school year. Further, the period between pretests and pdsttests is a fuli year, 
including summer, whereas the period between fall and spring testing includes only the 
school year itself. 

The magnitude of students' apparent achievement gains varies considerably 
depending on which of these testing schedules districts use (Keesling, 1981). Districts 
using annual testing cycles report achievement gains that are smaller than those of 
districts using the fall-spring schedule. To illustrate the difference. Figure 4.4 
superimposes test results from these two schedules. The figure indicates that second 
graders tested oh a fall-spring schedu' "or example, started their reading programs 
scoring at the 21st percentile rank and finished with scores at the 36th percentile 
rank, for an apparent gain of 15 percentile ranks (the dashed line in the figure). Yet 
those tested on an annual schedule (the solid line) moved from the 29th percentile 
rank to the 31st, for an apparent gain of only two percentile ranks. 

This figure indicates two important points. First, the amount of improvement 
observed in these students' achievement depends on the school districts' testing 
schedule. In both reading and mathematics, program assessments based on annual 
testing schedules yield much smaller achievement test score gains than those derived 
from fall-spring testing schedules. Second, many differences, especially in mathematics, 
appear to be due more to differences in starting achievement levels than 
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Figure 4.4 



Chan|[es jn Peittentiie _ 

Ranks for Reading and Matfiematlcs 6y Tes ting Cycle, 1983-84* 



Psrcentile Rank 
45 



46 

35 
30 
25 
20 
15 
10 



Reading 



Fall-Spring (9-month) testing cycle 
Spring-Spring (12-month) testing cycle 




^1 



1 1 



7 

Grade 



I - I l_J I I L_ 

9 10 11 12 



Percentile Rank 



Mathematics 




Figure reads: The percentile rank of the average second grader receiving Chapter I reading instruction increased 
from the 21st to the 36th using a M-sprihg^^^ while a spnrig-spririg testing cycle 

resulted In a change from the 29th to the 31st percentile rank for similar students: 



^Changes in percerrtlle ranRs ware calculated by first determining ail averages in normal curve equivalents 
(NCEs)j a staridamjized scale score metric^ and then cbhvertihg those averages to percentile ranks. See 
^^i>9ndix E M d©^ the rneasures jmd Appendix F for the comparable figure prfsented in NCfe The 

number of students Included in these analyses varied by grade level and subject; See Appendix G for details. 



Source: M. A. Carpertter ar)d P. A. Hopper, Synthesis of Chapter I Data: Summaiy Report Boston, VA: Advanced Technology. 1985. 
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in smM. achievement levels." That is. students tested in the fall demonstrate much 
lower entering achievement scores than those tested the preceding spring, even though, 
in several cases, the" demonstrate comparable scores at the end of the schocl year.^s 
ft is possible, of course, that these disparities reflect differences in the characteristics 
of the students enrolled or in the effectiveness of programs in districts employing 
these two testing schedules. But several other important possibilities have been 
suggested to account for these patterns. 

One hypothesis, presented by Keesling (1981). has to do with local scheduling of 
fall test administrations relative to the schedules recommesnded by test publishers. 
Student achievement improves rapidly during the first several weeks of the school year, 
presumably as students get back into the spirit of the academic enterprise, it is 
possible that, if districts tested students in the sixth week of the school year, rather 
than the first, for instance, much of the inflated portion of the fall-to-spring gain 
would be removed. V/ithoat inorc knowledge of the exact dates when districts 
administer their fall tests, it is not possible to know how much of the fall to spring 
gains are due to this early school-year phenomenon. However, we do know that the 
Chapter 1 Technical Assistance Centers work with districts to improve their testing 
schedules. 



test 



_^2KeesIing (i vX:; presented 1979-80 TIERS data showing T,ors compar ibis post- 
it. c ita then v/e j :nU in the 1983-84 data. 



*-Dc spite the remarkable differences in o«tcomes between test?i>f ^oh^dules the 
outc .u-ncs f-.ggestcd by the additional testing schedure shown in Figure' 4.4 reinforce the 
general pa,^rns described in Chapter 3. BotU teiidng cycles shown Figure 4 4 for 
instance, n-.. oate^thai older students enter reading programs at lower percentile ranks 
than younger students and that olc' r students gain relative! ]ess durj ^g the school 
year. Both estimates indicate higher relative starting rr.riks of students in mathematics 
programs, and both suggest that ma(.i:.matics gains ar<j groatc. smong younger students 
than among older students. 
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Another hypothesis^ also discussed by Keesling (1 98 1), is that districts may 
inadvertently inflate fail-tb-sprihg gains by the way they administer their tests. They 
may, for instance, be less careful in the fall because they don't perceive the fall test 
as important. Then, siiice the spring test measures student growth during the school 
year, teachers may work hard to prepare students for the spring test and to motivate 
students to do their best. In addition. Linn and others (1982) have listed a number of 
technicalities which could contribute to an artificial inflation of the fall-to-spring 
gains. 

Finally, there is the hypothesis that disadvantaged students gain less during the 
summer than other students do, and consequently lose their relative rank, if these 
summer losses were entirely recovered in the early fall of each school year, as 
Kef:Tliiig and others seem to suggest, then the patterns shown in Figure 4.4 reflect 
nothing more than an artifact of testing practices and momentary but inconsequential 
student forgetting. If, on the other hand, these summer experiences accumulate over 
time, contributing to a continual widening of the achievement gap between advantaged 
and disadvantaged students, then the patterns shown in Figure 4.4 reflect an important 
phenomenon regarding the learning patterns of disadvantaged children. 

Probably each uf these phenomena— technical details of test administration and 
students actually gaining relatively less in the summer—contribute to this pattern of 
test data. Whatever the reasons, the remark«>ble differences between the apparent 
achievement gains indicated by these two testing schedules does ssuggest that at least 
some portion of the Chapter 1 benefit is lost during the summer. 

Sustaining Achieve m ent Gains Acr^^ hddl Year s 

Concerns about the learning rates of disadvantaged students prompt two separate 
questions. First, what are the effects of continued services for those students who 
remain in the program for two or three years? Second, are the gains demonstrated 
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while students participate in Chapter 1 maintained after students no longer receive 
services? Each of these questions will be addressed in this section. 

Figures 4.5 and 4.6 present achievement scores of students who participated in 
Title I reading (Figure 4.5) and mathematics (Figure 4.6) programs for different 
numbers of years during which the Sustaining Ef iTects Study collected data. The lowest 
section in each figure shows the achievement profile of students who received Title I 
services all three years. In the upper two sections of Figures 4.5 and 4.6, achievement 
patterns are shown for students who partlcipsted in Titis I for one or two years, 
respectively^ of the three possible years. In M'^sc uppc- sections, separate lines ire 
presented for students who participated in TltJe I for different subsets of years. For 
instance, the top section displays separate growth patterns for students who 
participated during the first year only, the second year only, or the third year only. 
The middle section portrays students who participated during years one and two, years 
two and three, and years one and three. For each group of students^ the solid portion 
of the line indicates that the group participated In Chapter 1 at that time^ while the 
dashed portibn indicates non-participation, the achievement scores shown in 
Figures 4.5 and 4.6 were recalibrated each year. Under this method, the average 
achievement score is always 100, and changes in scores are measured relative to peers, 
just as changes in percentile ranks arc. thus, a strictly horizontal line indicates that 
achievement has improved at an average rate; there has been no relative gain or loss. 

Figures 4.5 and 4.6 indicate that participation is related to starting achievement. 
Students participating for only one of the years, regardless of which year it was, 
started with higher achievement scores than students who participated for either two 
or three years. Conversely, students participating for all three years tended to start 
at lower achievement levels, reiative to their peers, (starting scores in reading were 
81, compared to 85 for two-year participants in reading programs and over 90 for one- 
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Figure;; 45 



Reading tehievemeiit of Sludehts with Different ^tterns of Participation 
in Title I Acro^ Tiiree Wars, Tiie Sustaihihg Effects Study, 1976^ 



Standardized Achievement Scom^ 
1^ I Ohfr%ar Participants 
95 

9b 
85 
8b 
75 



Year 1 

Standardized Achievement Scores* 
■•00 1 — iWo-Sbar Participants 

95 

90 

85 

80 

75 



Year 2 



Year 3 




Year 1 

Standardized Achievement Scores* 
100 r— Three-Ybar Participants 

95 

9b 
85 
8b 
75 



Year 2 



Year 3 



Enrolled in Title I 
Not enrolled in Title 1 
Summer 



Year 1 



>fear 2 



Year 3 



Figure reads: Students who participated in Title [only during tlie first year of the study tjegan with ah average 
SOTre of 94 and ended their first year with an average score of 97. Their scores declined slightly 
acro^ the next two years. They completed the third year wit'i ah average score of 95. 

These are a form of standfrd scoresLfs descrited iri Appendix E, which have been standardized so that tlie 
tmm a. each Jest period equals ibb and the standard deviation equals 2a The number of students in ihese 
analyses varied grade level and pattern of participation; see Appendix G for details. 



Soacm-L. CarlBf, A Sludy rf Compsnsea>ry and Bemarmy Sdacatkm: The Sustaining Bfects Study, FinaJ Report. Santa Monica, CA: 
Sy^im DmeR^^meht Corjpbration, January 198a 
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Mathematics itehjevement of Students with Dlfferent^Pattems of Participation In Title I 
Across Three Ybaris, The Sustaining Effects Study, 1976^79 



Standardized Achievement Scores* 



100 




95 




90 




85 




80 




75 








OneAtar Participants 



Year 1 

Standardized Achievement Scorer* 
"•^^r" TWo-YBar ftrtlciparr- 

95 

90 

85 
80 



\fear 2 



VSar 3 



75 — 



1 



VSar 1 

Standardized Achievernent sicores* 
•1(30 t— 

Three-Y^ar Participants 



Year 2 



\fear 3 



95 
90 

80 



Enrolled in Title I 
Not eniolled in Title I 
Summer 



75 — 



Year 1 



V§ar 2 



\^ar 3 



Figure reads: Students >A^hQ participated ih_Title I during, the fiMye^^^ of 97 

and Hmshed the year an average of 101^ Thereate^ their scores declined and they completed 
the three-year period with an average score of 93. 



*These are a form of standard scores, as described in Appendix E, which have been standardized ^ that jhe 
mean at each test period equals 100 and the standard deviation, equals 20. The number of students In these 
analyses varied by grade level and pattern of participation; see Appendix G for details. 



SouiTce: L Carter, A Study of Compensatory and Elementary Education: The Sustaining Effects Study, Final Report. Santa Monica, CA: 
System Development Corporation, January 1983l 
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year participants in reading programs) and to remain at those low levels throughout 
the entire three-year period of data collection. Figures 4.5 and 4.6 also show that 
achievement scores rise while students are participating m Title I and remain stable or 
decline when students do not participate. 

With regard to our first question on the merits of multi-year participation relative 
to dhe-year participation, these figures indicate that three-year participants, who began 
with the lowest test scores in both reading and in matheiratics, appear to have done 
rib more than maiu^ain their relative positions throughout these three years. However, 
given earlier evidence that comparable low-scoring students actually lose their relative 
standing without special services, these horizontal patterns of test scores may indicate 
that Chapter 1 services have actually improved these students' achievement.^* The 
two-year participation data also do not yield a clear pattern with regard to 
accumuiat:ng benefits. For students who participated in reading programs during year 
one and two of the study, for instance, the second year appears not to add appreciably 
to what was accosnpHshed during the f ir^t year, in part because some of the first year 
benefits were lost during the summer. Those who participated in years two and three 
demonstrate a similarly ambiguous pattern this time because of summer time 
improvements. In contrast^ students who participated in mathematics programs during 
years one and two appear to have benefitted just as much from each year, and to have 
experienced no summer loss in between. For them^ the benefits of the second year are 
clearly over and above the benefits of the first yeai-. Those who p^trticipated in 
mathematics programs during years two and three demonstrate a similar accumulation of 
benefit. The Sustaining Effects researchers tried to test the effects of multi-year 

^^Figurcs 4.5 and 4.6 actuahy depict test-score perf^ rn^btr tnan percentile 

ranks. However, these tests scores were re-calibrated on each adininis^^?=ition, so that 
the average is always 10D. Gonsequcntly, the growth patterns can be interpreted 
analogously to those indicated by percentile ranks. That is, a horizontal line does not 
mean "no growth". It means "no change in relative standing". 
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participation for different grade level aiid subject matter combinations, but found a 
similar confusing picture (Wang^ et al, 1981). 

With regard to our second question, about the extent to which gains made while 
participating in Chapter 1 are retained once students leave the program, we rely on 
students who received service only during the first one or two years of the Sustaining 
Effects Study so that we can observe their achievement patterns during the school 
years foHowihg program participation; The one-year sections of Figures 4.5 and 4.6 
show that students who received services only during the first year of the Sustaining 
Effects Study gained in both subject areas while participating, but lost ground relative 
to their peers in later years. Students who had received first-year assistance in 
mathematics appear have lost mere in later years than those who received first-year 
assistance in reading. Because these students entered the study with higher scores to 
begin with, however, their achievement declines still did not leave them with scores as 
low as those of two-year or three-year participants; 

The middle sections of Figures 4;5 and 4.6 show that students who participate 
during the first two years, but not during the third, also gained during their 
participation, regardless of subject area; Those who received reading assistance during 
the first two years appeared to maintain their position during the third year, while 
those who received mathematics assistance during the first two years declined slightly 
during the third year. The middle portions of Figures 4.5 and 4.6 also indicate the 
progress of students who received services during the first and third years of the 
Sustaining Effects Study. These students appear to have benefited substantially from 
their first year of service, but they also experienced substantial summer losses 
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following their first year of participation.^^ Their scores were maintained during the 
second year, but they were again selected to participate in Title I during the third 
year. 

Generally speaking, these patterns suggest that students gain more than expected 
while they participate in Chapter 1, and in mathematics these benefits appear to 
accumulate across vcars. On the other hand, again in mathematics, students gain less 
than expected when they do not participate. The gains of previous years do not 
appear to help studen > rccdmmodate future learning demands. Most former Chapter I 
students maintained their relative standing in reading, but former Chapter 1 
mathematics students gained much more slowly without program assistance, and 
consequently lost more grouud relative to their peers. 

In a related substudy, the Sustaining Effects Study investigators also followed 
some of these Title I students over succeeding school years until i/ cy reached junior 
high school. They found that the earlier achievement increases from Title I services 
had virtually disappeared by the time the students reached junior high schbbi and that 
former Title I students enrolled in more remedial courses in junior high school than 
did comparable students. Thus, although elementary school Chapter 1 services 
positively influenced student achievement, they did hot improve achievement enough to 
eliminate students' need for further special assistance (Carter^ 1984). 

A more recent study^ undertaken by persdhhel of the Chapter 1 Technical 
Assistance Centers, under the direction of the Department of Education, (Gabriel, 
Anderson, Benson, Gordon, Hill, Pfaniiensteil^ and Stonehill, 1985), also assessed the 
sustained achievement of program participants. Their study relied on standardized 

^^Thesc summer pat . ns indicate only relative loss, not absolute loss. The scale 
re-calibrates scores on each testing occasion, so that the average score is always 100. 
Thus, a horizontal line indicates that students are gaining at an average rate, and a 
downward line indicates that students are gaining less. 
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achievement test data collected in spring 1982 and spring i983 in 17 school districts or 
State educational agencies. Analysts reviewed achievement patterns based on students' 
participation in Chapter 1 dnring ehese two school years^ with some students 
participating in Chapter 1 during both years, some during neither year, and some 
during the first year only or the second year only. Although not nationally 
representative, the study included over 66,500 second- through sixth-grade chiMren.26 

Student achievement patterns resulting from this analysis arc presented in 
Figure 4.7. The test scores arc re-calibrated for each test administreMon, so that 
normal growth yields a horizontal line. The four lines displayed in Figure 4.7 indicate 
student performance during the second year only. That is, thr first data point shown 
marks student achievement at the end of the first year. The patterns in Figure 4.7 
are similar to those reported by the Sustaining Effects Study: students who were never 
in Chapter 1 had higher, relatively stable achievement scores over time. Those who 
participated during both years had the lowest scores, though they showed small gains 
during this period of participation !n Chapter 1. Those who participated during one of 
the two years scored in between these other two groups. If they participated during 
the first year and not during the second, they exhibited declines during the second 
year- if they participated during the second year and not the first, they exhibited gains 
during the second year. 

The researchers responsible for this study hypothesized that there are three 
distinct populations of students whom school districts serve: (1) the general 
population, (2) a population needing some remedial assistance, and (3) a population 
heeding considerable compensatory assistance. The uppermost line in Figure 4.7 

— ^^hc study sample had slightly higher pretest scores (ranging from 1.4 to_43 
NCEs in rcajlihg and from 0 to 2.2 in mathematics) than those reported in TIERS for 
the overall Chapter ! population. This may be due to special features of the California 
Achievement Test which is used, or to differences in the populations served by 
districts participating in this study. 
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AcHievertieflt of StUdehts by Patter?^ of f^rticlpatioh 
In ehapter I Acro ss TWo Wars, 1982-83 
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Figure reads: From the spring of 1982 to the spring of 1983, students who received no Chapter I servicss in 

either school year had average NCE scores of slightly less than 6&, and demonstrated slight gains 
between spring 1982 and spring 1983 in both reading and mathematics. 



*NeE scores are transformations of percentile ranks to a standardized equal-interval scale; see Appendix E. 



Source: R. Gabriel, B. Anderson, G. Berispn, S- Gordon, R. Hill, d. Pfennenstiel and R. Stonehill, Ihe Susmlned Achtevermnt of Chapter I 
Students. US, Dept. of Education, January idsa 
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represents the achievement growth of the first group; the middle t\v ' i .? ' present 
the second group, and the lowest line represents the third group. The group tStat 
received continuous compensatory education appears to reap the smallest 5 h evement 
benefit; yet we cannot know what their achievement patterns would have been in the 
iaibsence of Chapter 1 assistance. 

Finally, recent lohgitudlna! :M3iyses of the achievement levels of former 
Chapter 1 participants In Columbus, Ohio (Amorose, Biown, Duffy, Morgan, and 
Thompson, 1986) and in St. Louis, Missouri, and Lincoln, Nebraska (Pfannensteil, 1986) 
compared the fourth- and fifth-grade achievement scores of students who had received 
Chapter 1 services for one, two, three, or four of the preceding years. These data 
indicated that many groups of students were able to retain standardized achievement 
scores at approximately the levels they had at the close of their Chapter 1 services. 
However, students who received three and four years of service tended to have 
relatively lower scores even when they left Chapter 1 and had still lower scores when 
retested later on. 

Long-Ttp m Program Effects in Areas Other 
Than Achievement Test Scores 

Recently, several analysts and policy makers have taken an interest in other 

indicators of program benefits, such as long-term patterns of school attendance, grade 

retention, graduation rates, and even future educ-:tion and career accomplishments: 

Assessing long-term program effects would require researchers to follow a group of 

students for several years. Even if a study were able to follow a sample of students, 

It would encounter interpretive problems in attempting to account for differences. The 

longer the period of time between the receipt of services and the measurement of 

outcomes, the less able analysts are to attribute observed changes to participation in a 

program: too many things have happened in the meantime. 
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While such long-term studies have hot been conducted in the context of 
Chapter U they have been conducted for preschool programs. The evidence suggests 
that, while achievement-score gains tend to fade away over timci other important 
outcomes, such as school attendance rates and grade retention rates tend to remain 
(Smith, 1986). Tnc discrepancy between these long-term behavioral patterns and the 
long-term achievement pattern make the data difficult to interpret 

In an effort to disclose the long-term education experiences of former Ghapter 1 
students, the National Assessment of Chapter 1 funded several small projects to analyze 
existing State and local data bases. Most were unable to follow up on students for 
more than five years, and most concentrated on long-term achievement patterns. 
These studies indicated that lower-scoring pupils were more likely to receive multiple 
years of Chapter i services (Amorose, et al., 1986; Pfannensteil, 1987, Kirshsteiu, 1986; 
Plato, et al., 1986) and that they were more likely to receive services from other 
categorical programs as well (Plato, et al., 1986; Pfannensteil, 1986). Where data on 
grade retentions were available, they indicated that Chapter 1 participants were 
retained in gr?de more often than non-participants (Amorosc, et al.^ 1986). Former 
Chapter 1 students in Montgomery County, Maryland were less likely than other 
students iii that district to pass the State*s ninth-grade functional reading and 
mathematics tests, and more likely to have been suspended in the past three years. 
These data suggest that, even after leaving Chapter 1, these students are likely to 
continue to have problems in school. 

Summary and Discussion 

In this chapter we reviewed evidence regarding the growth of disadvantaged 
students* achievement over the summe. and into subsequent school years. The evidence 
indicates that the achievement test scores of disadvantaged students tend to decline, 
relative to those of more advantaged students, as they progress through the grades, 
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aiid that the achievement gap widens during each summer. Chapter 1 assistance during 
the school year raises the relative achievement ievcls of some of its students and heips 
others maintain their relative position. Once students leave Chapter 1, their scores 
again decline. Because participation in Chapter 1 is related to achievement levels at 
the outset, it is difficult to estimate what would happen if students continued to 
receive services over many years. Our principal findings are as follows: 

7. The afcHtevemeht gab between di^advahtagcd and advantaged students appears 
to widen during the summer months. 

Though few studies have been designed specifically to monitor the gap between 

the achievement levels of advantaged and disadvantaged students, available data sources 

indicate both that the gap widens as students move throujgh the grades and that the 

summer mbnthc contribute to the increase. By the time students are in sixth grade, 

Ibw-achievers are substantially behind their higher-achieving counterparts. Further, 

the amount studeits gain during the school year is much less, relative to the 

diffen :^ between high- and low-achieving students, than gains made by younger 

studeh'i . 

8. Title I - subbbrt e d si Hn^met- programs ha^ve hot narrowed the achievement gap; 
howevef, 4h<^^fbgi^ms-of ten were no to be academicailv 
rigorous. 

In earlier periods of Title I histbry, summer school programs were a popular 
option^ in part because it was believec' .^t these might prevent the achievement of 
disadvantaged students from falling even farther behind. Though the evidence suggests 
that such programs did not substantially alter student achievement, it also indicates 
that these programs were not designed to be academically rigorous. Further, they 
usually lasted only six weeks. Thus, the extent to which such programs may benefit 
disadvantaged students is still not known. 
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9. Students who disc^frtiniie ^Fitfe^^^ mr^rad^ual^hf 4d fese^round when thev 
no longer receive services. 

When Title I was first legislated in 1965, liiany observers had assumed that a year 
or two of compensatory education services would be sufficient to alter the educational 
futures of disadvantaged children. The data suggest that such long-term benefits do 
not occur. There are at least three hypotheses that can account for these findings. 

One hypothesis is that gains appear not to be sustained because the content 
children learn each year— and the content that achievement tests measure each year- 
differs from the preceding year's instructional content. Ghapter 1 may help children 
master their seconU-grade material, for instance, and children miay sustain that mastery 
throrgh future grades. But second-grade assistance would not help children master 
their fourth-grade material. Under this line of reasoning, children are in fact 
sustaining the gains they made while participating, but the achievement tests of future 
years no longer measure that content. 

A second, and related hypothesis is that disadvantaged children continually learn 
at a slower rate, and that they consequently continue to need more instruction and 
repetition in order to learn and to retain what they have learner- This hypothesis 
implies that most Chapter 1 students would need to receive services every year, from 
kindergarten through twelfth grade, not just for one or a few years. 

A third hypothesis is that Chapter 1 students heed help learning to learn. This 
hypothesis suggests that it may indeed be possible to provide a short-term 
supplementary service which will have a long-term impact. But the nature of that 
service would be quite different from what is now typicaUy provided under Chapter 1. 
In fact, under this hypothesis. Chapter 1 instruction which focuses on curriculum 
content will necessarily need to be repeated year after year, for it does not teach 
students how to learn front the regular program. 
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16. gfaapter i students with very jow achievement ievels aDDear to maintain their 
relative achievement levels while participating in Chapter h but not to move 
ahead. However, evidence suggests thev would have lost ground relative to 
their peers if thev had not received services. 

The evidence available suggests that the lowest-achieving students receive multiple 
years of service, and that, while their achievement scores rise from year to year, 
they do not rise enou' 'i to substantially alter the students' relative academic standing. 
Thus they appear to continue on at relatively low achievement levels. 

However, because the learning curves of Idw-achieVing students differ from those 

of higher-achieving students, it is difficult to estimate the extent to which, if at all. 

Chapter i services have benefitted their lowest-achieving participants. Evidence 

presented in the last chapter suggests that these students could have fallen farther 

behind their peers if they had not received Chapter i services. If this is so, then 

what appears to be a lack of impact, because students retain their relative position, 

may in fact represent ah impact in that students have not fallen farther behind. 

11. No riatibhallv-represehtative stu dies have examined the long-term effect nf 
Chapter 1 pr<>gr4TT ^ H>i^^faauatf<^^ edu cation, or adult literacy. 

An assumption underlying the Title I and Chapter 1 legislation is that 

compensatory education will help poor children improve their future educational 

prospects. While research indicates that some Chapter 1 students continue to 

(experience a range of difficulties as they proceed through school, information about 

the long-term effects of participating in Chapter 1 programs is not available. Studies 

of these effects would be difficult to conduct and to interpret if they were conducted, 

both because participation in Chapter 1 can vary greatly from student to student, and 

because it would be difficult to keep track of former ."^tudenis and their educational 

experiences over a long period of time. 
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5. Project Characteristics and Student Achie /emect 

Gentra4 Fi ndings 

12. Large-scale studies designed to identify particular project 
characteristics that improve student achievement test scores have 
yielded inconsistent or inconclusive findings. 

13. Researchers have recently identified a number of instructional 
practices that are iikely to increase the achievement of 
disadvantaged students and that can be used in Chapter i 
programs 

Chapter 1- arid Title I-suppbi ted coriiperisatory education services have been 
provided to disadvantaged students for over 20 years. Throughout that time period, 
questions have been continually sed abo?it the value of these prograriis. Prograrii 
critics point to the over $3 billion spent each year for these services, arid argue that 
the gains we described in earlier chapters are not sufficient to juijtiiy that cost. 
Program advocates point to the serious need that disadvantaged children have for 
educational help. Most of the studies we reviewed in earlier chapters were motivated 
by these debates and have contributed to them. Evidence on iong-tcrm program 
effects and on the learning rates of different kinds of children suggests that the 
problems of educational disadvantage are riiuch riiore difficult to solve than the original 
designers of Title I legislation had assumed. Eviderice di one-year program effects, on 
the other hand, suggest that these services do noticeably alter students* achievement in 
mathematics and reading. 

Even more problematic, at least in regard to this debate, is the apparent 
variability of program effects across subject matters, grade levels, types of student 
served, and localities. Such v nability suggests that it is at least hypothetically 
possible for Federally-supported services to have a greater average impact than they 
now have. If services could be improved at the lower end of the distribution of 
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student achievement, so that they emuiarad the upper end, perhaps the net impact of 
the legislation would be increased; 

In ah effort to improve the effectiveness of Chapter 1 services, researchers and 
practitioners have launched many efforts over the last 20 years to identify effective 
compensatory education practices. They have sought such practices primarily in order 
to assist other, less effective projects. Efforts to spotlight effectsvc practices have 
taken three forms: (1) identifying effective Chapter 1 pro Sects and informing othcv 
districts about therh, (2) identifying specific project features tnzt seem fco promot*- 
student achievement and that couid be implemented in other projects; and (3) 
identifying effective practices in general education that could be implemer.ied within 
Chapter 1. This chapter reviews each of these efforts. 



The earliest Federal a^^empts to improve Title I programs concentrated on 
identifying arid describing ii projects that demonstrated exceptional achievement 
increases, in the hope ths othen chool districts could adopt these projects and 
th eby increase the effectiveness of t^tei/ own programs. hese efforts prompted 
development of a dissemination network and publlcatiou of an annual catalogue of 
effective practices, called Education Programs that Work. The annual catalogues 
described effective projects supported by other Federal categorical programs as well as 



This method of dissemination proved popular, in part because of the recognition 
that local project developers received. However, its success as a mechanism for 
Improving Chapter 1 projects has been miixed, for two reasons. First, it was not 
possible to know which aspects of effective projects are responsible ior students' 
achievement gains. A project could be successful because of the way it involves 
parents, the way it uses teachers or teacher aides, the way Chapter 1 services are 




Title 1. 
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coordinated with the regular school program, or for other reasons. Irideedi its success 
may be the result of one talcnrcd teacher; Second, as dissemination of effective 
projects became more commonplace, it became clear that districts did not adopt cr ^m 
in their entirety. Instead, districts adapted them to fit local circumstances. The 
combined effect of, oh the one hand, researchers disseminating projects whose most 
important features could not be idcntifi-id, and of, on the other hand, districts adapting 
th'jse projects to fit their own circumstances, raised questions about the ultimate 
effectiveness of this improvement «?trategy. For districts could unwittingly alter the 
very feature that contributed most to student achievement, and no one would know it. 
Thus, though the recognition of effective projects provides an incentive for projects to 
do well, it is less useful as a strategy for improving the effectiveness of other 
projects, in part because such recognition of effective practice does not improve our 
knowledge of what contributes to effectiveness. 

Ef'f^i^^<> Ideirtji^/^Eff^ctiv e^F^ of Chapter i Projects 

One strategy for increasing our knowledge of what contribute: lo effectiveness is 
to study a wide range of projects, not just effective projects but ineffective projects 
as well, and to see how these projects differ. Three national data sources have been 
used to study the relationship belwecn different project features and gains In student 
achievement: the Instructional Dimensions Study, the Sustaining Effects Study, and 
TIERS. 

The instructional Dimens ions Stadv 

To sort out the relative benefits of particular project features, NIE*s Instructional 
Dimensions Study (NIE, 1976; Coolcy and Leinhardt, 1980) developed a model of the 
instructional process which focused on five factors: student motivation, opportunity to 
learn in the classroom, degree of individualization in the program, the nature of 
instructional events, and the characteristics of project teachers. These factors were 
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further broken down into a number of specific descriptors. Using these descriptors as 
the basis for tneir data coliection, researchers observed 400 first and third grade 
classrooms in 14 school districts, testing students in each classroom. Their goal was to 
estimate the contribution of each instrucdonal factor to student achievement. 

The Instructional Dimensions Study yielded thre<; important findings. First, it 
fb;»nd that individualized instruction did not make a difference in studcr achievement. 
That is, students gained approximately the same amount whether they v/erc taught 
individually or in groups. This finding was particularly important when it was 
published in 1976 because Congress was considering whether to require individualization 
under Title 1. The Instructional Dimensions Study defined individualization to mean 
that: (1) specific learning objectives were assigned to individual children, (2) children 
were taught in small groups or were individually paced, (3) children had the benefit of 
individual diagnosis and prescription, or (4) there were alternate learning opportunities 
and sequences available for ihdividurl children. 

Second, it found that pu 1 Idu t progi^affls were a more effective instructional 
arrangement in some circumstances, and f f^'elass-^ osrams were more effective in 
others. First-grade students seemed to benefit more from in-class programs, whether 
the subject ^^iis reading or mathematics. Third-grade students, on ihe other hand, 
derived more benefit from pullout programs in mathematics and showed no differences 
in reading. The extent to which pullout and in-class configurations may have yielded 
other patterns of effects in other grades couiw not be determined. Because of ongoing 
debates about prograni structures, this finding coh'^inues to be important today. 

The third important finding from the Ins ructiohal Dimensions Study was that the 
fsotor called opportunity to learn was the only one consistently and significantly 
associated with student achievement. This factor included such program features as 
the amount of time devoted to instruction ard the size of the groups in which children 
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were taught. Hdwevcfi it aiso included ths extent to which the content of the 
achir-'ement vu t matched the content of the Ghapter I curriculum, and further analysis 
indicated that this content over] ip between test and text was the most important 
aspect of students' "opportunity to iearn."^' 
The Sustajning Effects Study 

The Sustaining Effects Study also examined the relationship between classroom 
instructional practices and student achievement. Unlike the Instructional Dimensions 
Study, the Sustaining Effects Study gathered data on instructional techniques from 
teacher questionnaires rather than direct observation. Analysts in this -tudy assessed 
the relative contributions of many academic and nonacademic factors o students* 
achievement and tested a number of models of the educational process. In each case, 
the outcome of interest was student achievement scores vGarter, 1983). 

The Sustaining Effects Study researchers also found evidence of a relationship 
between in^tryptign^f ij^ and achievement (Wang, 198i).. ^1?^ the reiationship did 
hot appear in every grade and subject examined. Furtht. .. . linc spent specifically 
with a special program teacher or in a special setting did not have appreciable effects 
on achievement grovs^th, though time spent in regular instruction did. Other factors 
that appeared to make a dif f erer.ce were the teachers' years of experience, the 
frequency with which students received feedback on their performance, and the lack of 
disruptioE in the classroom. Sustaining Effects Study researchers concluded that the 
amount of compensatory instruction Would not bv-itselF close the achievement gap 
between compensatory education students and their "ondisadvahtaged peers and that 



This does not necessarily mean that teachers were or should be "teaching to the 
test." Content overlap means that, for instanciSi if teachers are teaching Roman 
numerals in their mathematics class, the test includes questions oh Roman numerals. 
Or, if the reading textbook includes a large nature vocabulary, so does the reading 
test. Freeman et al( 1983) have Si',cwn that content oyerlao varies substantially across 
different combinatio:;- oi textbooks ^nd standardized tests. 
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more information was needed about the components of compensatory instruction to 
know which are most important to achievement. 
TIERS 

TIERS included a reporting Form which States could use to describe specific 
project characteristics, as well as student achievement gains, in grades 2, 6, and 10. 
Project characterist'cs included the number of days the typical student participated in 
the Title I program, whether the program was offered in clasK or on a pullout basis, 
and the studeht-to-teacher ratio. 

For the most part. State educational agencies* reporting of programmatic data has 
been sporadic and prone to error, more so than has been true of the TIERS 
achievement data. Analyses of the data have yielded no clear patterns relating project 
characteristics to gains in student achievement (Wood, 1984). 

Analysis o^^forts to Identifv^f fective Project Features 

Despite elaborate and sophisticated efforts to identify effective project features, 
only a few such features have been found. Four important limitations account for 
these results. 

The fir5. „.rerns the ability of researchers to know in advance which features 
of aii educational program are likely to be significant and should therefore be 
documented in their stady. The Instructional Dimension Study researchers relied on p 
theoretical model of the instructional process to decide './hat to observe and then 
observed actual practice to document the factors they expected to be relevant. The 
Sustaining Effects Study researchers relied on existing student records on teacher 
questionnaires for their evidence of practice. In both the Sustaining Effects Study and 
TIERS, the characteristics that tended to be documented were those that could be 
efficiently observed and m»i::isured (e.g.^ whether the teacher has an aide, the teacher's 
years of experience, whether instruction occurs within & ^ular classroom or in 
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another setting). Yet what may be most significant is not the presence of a classroom 
aide but the training the aide has received or the amount of time the teacher spends 
discussing each day's instructional goals with the aide. Similarly, the physical location 
of services - pullout versus in class, for instance - may not be as important as the 
character of the services provided. When the Instructional Dimensions Study was 
completed, the NIE convened a group of teachers who had participated in the study to 
discuss its findings. In that conference (NIE, 1978), teachers described a wide variety 
of practices that were all labeled "pullout" but that had remarkably different 
instrucHonal meanings. They argued that the distinction between pr^^out and in class 
configurations was hot as instructionally important as these other variations were. 

Secdrid, this line of research is complicated by the fact that Chapter 1 services, 
while defined as if they were entirely separate from the rest of the schco! program, 
are not separable in their impact. In most cases. Chapter \ - -^s are provided 
during only a small portion of the school day-for half haps. During the 

rest of the school day. Chapter 1 students receive th: :;ai auction as other 
students do. And their achievement is influenced by both sets of instruction. The 
Instructional Dimensions Study concentrated only on the Chapter 1 portion of the 
school day, and ??ttempted to determine aspects of Chapter 1 services that were related 
to gains in student achievement. Yet at least some of the students' achievement 
growth must have been due to the regular program. To t :e extent that student 
achievement is also influenced by the regular program, ;he re nrcixcr has more 
difficulty separating out important features of Chapter 1 projects. The Sustaining 
Effects Study researchers included measures of both Chapter 1 and regular instruction 
in their analysis. When students received instruction from multiple teachers, the 
"practice" was defined as the average of these teachers* responses to the questionnaire. 
Consequently, when these researchers identified a factor such as "lack of disruptions in 
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the class** as relevant to $*a: .i^ cn.eht., they ^vere referrinjg to a factor that 

matters regardless of whetbe? it u :raiv ia Chapter 1 or in rhe regular s.:itting. 
Conseqaentiy, their strategy also im kcs : *1ifficuU to idcnnfy the important features 
of Chapter i. 

Third, these studies have tended to search for the unique effects of each 
project's characteristics, even though in f«iracticc these characteristics operate in 
tandem to influence student achievement. For example, classroom aides may contribute 
significantly to student achievement in one project and yet contribute little in another 
project. The difference in the salience of their roles may be related to other project 
features such as the curriculum or student grouping practices. Yet because the 
contribution of classroom aides varies from one project to another, their average 
impact can appear to be nil. The number of ways in which these instructionally- 
relevar.t project cLdracteristics may be combined exceeds the capacity of existing data 
to 'dehtify, define, and assess these variations. As we have seen, however, attempts to 
identify whole projects that are effective have been equally unsatisfying. 

Finally, these analyses are complicated by the variety of relationships that can 
exist between project features and stwdent needs. "Chapter i children" do not 
constitute a homogeneous group, and the features of local Chapter 1 scrvsccs covld 
purposely vary in response to student needs. In our first report, we showed that the 
pdpuls^tion considered to be educationally deprived can vary a great deal from school to 
sciidol because the overall populations served by these schools differ. Further, even 
within a given school or district. Chapter 1 children are not a permanent subset of the 
student population. Instead, they come and go as their achievement fluctuates relative 
to that of their peers. These variations in populations served, combined with 
variations in the characteristics of Chapter ! services across schools and districts. 
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make it extreriet> difficuit to determine whether a particular kind of services is 
generally ef ftcaojous; 

EffettSnfaJ : :^ntifv Generaiiv Effective Praetieea 
NtH Li mited to Chapter 1 

Large-scale studies designed to identify effective features of Chapter 1 projects 
have increasesd our knowledge more than have project recognition efforts. Bat these 
efforts are large and expensive undertakings, and the knowledge gained has not been 
sufficiently rich or informative to guide local practice. Two recent efforts to improve 
bur understanding of the relationship between practices and student outcomes draw 
more oh existing research from education in general, rather than confining themselves 
to the Chapter 1 context. These efforts have sought to identify educational practices 
generally recognized as effective, whether or not such practices are used in Chapter I. 
One of these was conducted by the Education Department as a program improvement 
effoirt. the other by the National Assessment of Chapter 1. specifically for this report. 
EB^Kfnih^.f,^ fif gfl^t^r^ PromntincF Chaptei^l t^^f w^m^ 

ED'S most recent prograS n^v-ovcmt^iti effort in Chapter 1 incorporates both the 
project recognition approach and the effective-characteristics approach. It began with 
a review of research oh thi characterisijcs of «T«^ctiv^ schools. In the past 10 years 
or so. a substantial body of research has accrmu/attd regarding the features of 
effective schools, and several researchers have attempted to synthesize these findings 
and to develop a list of school characteristics that appear to make a diffcreiicc. ED 
capitalized on these findings and on researchers' inte.r?^-*s in codifying the findings. It 
created a list of characteristics that were particularly relevant to Chapter 1 projects. 
ED'S list included such organizational attributes as a "positive school and classroom 
climate-' c :d 'parehtal/commMnity involvement"; and it included such instructions! 
attr ibutes ai- the "use f ac;adem?c learning tima". Using these characteristics as s. 
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guide, ED then asked State educational agencies to nominate Chapter 1 projects that 
exemplified these features, iaihd asked the nominees to describe how their programs 
incorporated these ideas. From these documents, ED identified 113 Chapter 1 projects 
as being particularly effective. Finally, ED described, on one ' :he school 
characteristics linked in the research literature to higher ave r:^, ^:. achievement scores 
and, on the other, its examples of Chapter 1 projects which implemented these ideas. 
ED produced a two-volume ^ ^ : (Griswold et al., 1986) which has been 

disseminated to Chapter 1 pr : - -.cross the nation as a guide to improving project 
operations. 

This approach combines the best features of the project rf^cognitibn approach and 
the effective-features approach. With these descriptions of specific projects, other 
projects are given concrete examples that they can emulate. In addition, the 
effectiveness of the projects is analyzed, for they are defined according to the 
features that ED expects, based on research to be important. Finally, the research 
findings regarding these important features is also summarized, so that adopting 
projects may learn the principles of effective practices that are exemplified by the^e 
projects. Thus, if they choose to adapt the projects to suit their local circumstances, 
they rnay do so while still honoring the important principles underlying the project. 

One difficulty with this approach, however, is chat the effective practices were 
derived from research en schools, not on Chapter 1 projects. Some of the findings 
from this body of research may be particularly relevant to Chapter 1 projects while 
other ndings may not be. Academic learning time, for instance, is something that 
many researchers have found to make a difference in a variety of settingij, and is 
likely to i.:atter in the context of Chapter 1 as well. But other school f^-^tures 
identifie-'^ by that research, such as a positive school climate nnd a set of goals that 
are shared by faculty aiil administrators witfiji the school may not transfer to 
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Chapter 1; School ciimate and shafied instructional goals bot'a refer to the culture of 
the school as a whole, not to particular programs within the school, if Chapter 1 
programs achieved these features while the rest Of the school iiiS noi, : rtresumsd 
benefit to student achievement may not occur. In fact, some researchers (e.g.. Smith, 
1986) have argued that because Chapter 1 progranss involve only a subset of teachers 
and a subset of students within each school, they in«y actually encourage separate 
subcultures with separate educational goals, thus decreasing the schoors ability to 
develop the sort of climate now recognized as important to student achievement. 
Thg National Assessment's ReView^o f Effe c tiv e £du5atjm . ^i Practices 
The National Assessment of Chapter 1 asked a hUmbsr of researchers to review 
the research evidence with which they were most fsmiJiar and to describe the 
Characteristics of sound educational practices thai could Or would be appropriate for 
disadvantaged children.28 Researchers were asked to review evidence about effects of 
practices in five areas (Williams et al., 1987): 

• Chapter 1 school and student selection procedures; 

• Chapter 1 program design and staffing structures; 

• Relationships between Chapter 1 and regular programs; 

• Tarent involvement; and 

• Curriculum and instruction practices. 

These five topics consider Chapter 1 both in isolation and in conjunction with the 
regular school program. The first two topics address specific features of local 
Chapter 1 projects, and most of the research reviewed on these topics came from 
evaluative studies of Title I and Chapter 1. The second two topics-relationships 
between Chapter 1 and the regular prograr.^ arr? parental involvement-have more to do 



e 'fThesc researchers were not asked to address ;he political or fiscal implications 
of their findings but instead to state their intejpretations of current research fin<f ngs 
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with the child^s entire educatibnai experience^ and where and how Chapter 1 services 
fit into the overall configuration. Finally^ the last topic--carriculum and instructional 
practices— addresses issues that are almost completely independent of Chapter 1, though 
the findings in this area could be implemented within Chapter 1 as well as in any 
other setting. 

The researchers who addressed the issue of Chapter 1 sehodl^^Hi^ti^ent 
selection discussed our first interim report to Congress at lengthy and suggested that 
the provision of services only to the lowest-achieving students within Chapter I 
schools may not be the best way to allocate se> vice. Rather, these researchers 
suggested targeting Chapter 1 funds on those schools serving the highest 
concentrations of iow->inc6me students, an^! '^en serving all students within these 
schools. 

There were two reasons for this suggestion. The first reason was that Chapter 1 
now serves many students attending schools which do not serve unusually high 
ccncertrations of children from low-income families. These researchers thought that 
cdncc^ exiting funds in fewer schools would result in the provision of services to a 
more needy group of students. In bur first report to Congress^ we showed that 
students attending schools with high coiiCentratidn of poor children were doubly 
disadvantaged: even non-poor students attending such schools are more likely to be 
inw achievers than ai^. .^Gii-pbor students in other schools. The second reason for this 
suggestion, however, had to do with the research relating to effective schools. One 
researcher (Smith, 1986) argued that concentrating services in fewer schools offered 
the additional benefit of enabling schools to develop schoolwide projects^ thus 
eliminating the potentially divisivr; subcultures liow hypothesized to exist in Chapter 1 
schools; and that concentrated services and schoolwide projects would also elimfnate 
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cumbersome withiu-school accountability mechanisms ihat now accompany Chapter 1 
funds. 

The evidence suggests, that concentration oF funds would indeed increase the 
number of poor and low-achieving students served. However, the relationship between 
school and student selection decisions on one hand, and program quality on the other, 
is not clear-cut. If the Chapter 1 funds in a schoolwide project were used to add a 
science teacher to the faculty, they m^y .-^^t increase the students' academic learning 
time. On the other hand, if T were !>?cd to train all teachers in the school about 
the special educational need: .dsadvaritaged students, to facilitate an organizational 
climate and a set of high expectations for student learning, then perhaps such an 
alteration would positively benefit students. 

With regard to program and staffinp structures , researchers expressed frustration 
at the lack of knowledge about siich basic practices as serving students within their 
ow:; classes versus dulling them out of these classes to receive s. -ices in another 
setting. As we pointed out earlier, the educational setting may be less important to 
student learning than is the instruction that occurs within the setting. One esearcher 
concluded: "What is disconcerting. . . is that the proponents of puHout ha\ ^ 
comparatively little evidence to justify its widespread use and the critics of pullOTn 
would be similarly hard-pressed to justify a wholesale shift to in.cl?.ss instruction" 
(Archambault, 1986 p. 2). 

Another program and staffing feature that h?: considerably in the 

past is the ratio of students to teachers. Most of cct^i ...nv. . -jts in traditional 
classrooms, with class sizss ranging frrm 20 to 35 students, depending on grade level. 
The available evidence suggests that differences in this range do not substantia! v 
influence student achievement. However, the researcher at this conference who 
addressed this issue felt that group sizes as low as those adopted by Chapter 1 
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programs— often as few as 10 students in a class-dd make a difference (Gbopen 1986). 
Thus, one aspect of Chapter i that could be cdhtribtitihg to student achievement is the 
reduced student-teacher ratios which it supports. Such small groups may enable 
Chapter 1 teachers to provide students with the individualized help and attention they 
need. 

Yet these ratios are only provided for a short period each day. For the bulk of 
the school day Chapter 1 students are in regular classrooms and share their regular 
teachers with 30 other students. This fact raises questions about the role ChaiDter 1 
services play in the entire school program, a topic also addressed by conference 
participants. 

Two important points were made about the r^^atibnship^^tweeh~CharDtei^+^iid 
the rcig iria^^^Qgram First, Allingtbh and Johnston (1986) argued the importance of 
'*curricular congruence** for improving student learning. Second^ Griffin (1986) argued 
the importance of coordinating iiiservice training for teachers. Both pointed to 
evidence of shared instructional goals as an important feature of effective schools. 
Griffin pointed to evidence that staff training is more effective when it is provided to 
ail teachers in a school, rather than to subgroups such as Chapter 1 teachers, and 
Allihgtoh and Johnston pointed to evidence that learning is impeded when students 
receive inconsistent lessons. When teachers do not coordinate, the responsibility for 
finding connections across lessons falls on the students, and students who are having 
difficulty in school to begin with have a particularly difficult time when they must also 
take responsibility for cdbrdihatirig their own ihstructibh. 

With regard to eaiH^n^ i^iv^tfvem^nt . there was consensus among these researchers 
that student achievement improves when parents take a strong interest in their 
children's education and encourage them tb do well (McLaughlin and Shields, 1986; de 
Kanter, Ginsburg and Milne, 1986; Rich, 1986). Such attitudes may manifest themselves 
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through sttch behaviors as helping children with their homework or attending parent- 
teacher conferences; Some forms of assistance need hot rely on high levels of parent 
education: turning off the television when it is time to do homework and providing a 
place for students to work can convey the importance of these activities to students. 

Finally, with regard to curriculum and instrii ctionfil hrna M^ researchers 
identified a number of instructional factors which, taken together, increase students' 
opportunities to learn.29 One researcher summarized the evidence as follows: 
achievement scores are improved when teachers "emphasize academic instruction, expect 
their students to master the curriculum, and allocate most of the available time to 
curriculum related activities" (Brophy, 1986 p. 8). Other researchers not participating 
in this conference have also pointed out that time devoted to instrsaetion makes a 
difference to students' achievement, but that time alone is not sufficient to raise 
achievement. The time must be used well (Karweit, 1983; Walberg and Frederick, 1985; 
Cooper, 1986). Our researchers also argued that when students are actually engaged in 
academic work, rather than getting organized or moving from one activity to another, 
and when teachers teach actively, rather than assigning large amounts of seat work, 
students learn more. 

Researchers disagreed, however, on the extent to which teachers should alter 
their behaviors when working with disadvantaged students. One review indicated that 
disadvantaged students "need more control and structure from their teachers: more 
active instruction and feedback, more redundancy, and smaller steps with higher 
success rates. This will mean more review, drill, practice, and thus more lower-level 
questi bhs" (Br ophy. 1986 p. 35). Brophy also acknowledged that "across the school 

_ 29q„e reason this Tactor may have been significant in these studies, but not in the 
earner large-scale studies described above, is that these researchers were able to code 
subtler and more instruetionally-relevant aspects of opportunity to learn These 
resea,rehers agreed, for instance, that the time allocated to instruction would not by 
itselfi increase the time students spent actually engaged in learning. 
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year, it will mean exposure to less material^ biit with emphasis on mastery of the 
material that is taught and on moving the students through the curriculum as quickly 
as they are able to progress'' (BrdphVi 1986 p. 35). 

Other researchers (Romberg, 1986; Calfee, 1986) argued, however, that such 
special treatment of disadvantaged students could translate into lower teacher 
expectations, less material covered, a more boring educational experience for students, 
and, ultimately, lower achievement. These authors argued that an exclusive focus on 
basic skills, especially in later grades, would not allow students to go beyond these 
skills to grasp important concepts and to think and solve problems. 

S4Hnmarv and Implications for Chapter 1 

A number of important findings emerged from this collection of research reviews. 
Some of these findings suggest hypotheses for why Chapter 1 services have the 
outcomes they do. The research on student-teacher ratios, for instance, suggests that 
Chapter 1 may facilitate learning by providing small class sizes, even though these 
smaller groupings occur for only a portion of the school day. On the other hand, 
research on the features of effective schools suggests that Chapter 1 may also hinder 
student achievement by restricting the school's ability to create the shared academic 
goals, high expectations, and strong achicvcment-oriehted school culture that arc now 
recognized to be important to student achievement. These findings also suggest that 
disadvantaged students might Icarh even more if, for instance, the sizes of their 
regular classes were reduced substantially, or if Chapter 1 teachers were more fully 
incorporated into the schooPs overall instructional program. 

Other findings indicate that the features of Chapter 1 programs that have tended 
to capture the attention of policy makers-features such as pullout and in-class 
programs and individualized iristructiori-are not particularly influential on student 
achievement, and should probably not receive the amount of policy concern that they 
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traditionally have received. Still other findings point to aieas outside the influence of 
Federal policy makers-the curricaiuin, the teachers' instructional strategies^ the 
effective use of learning time in classrooms, and the culture of the school as a whole. 

One feature of Chapter 1 for which these findings are relevant is the 
supplementary character of Chapter 1 services. The research reviewed here indicates 
that the more opportunities students have to learn, the more they actually learn, and 
Chapter 1 prrgrams are generally designed to provide students with additional 
instruction in reading and mathematics. But as a supplement to the regular program. 
Chapter 1 services usually represent only a portion of a child's total instructional day. 
The most recent evidence suggests that Chapter 1 services average 44 minutes a day 
for four days a week (Advanced Technology, 1983). Further, Chapter 1 services may 
not actually provide additional opportunities to learn, for two reasons. First, students 
usually receive Chapter 1 services during the same six-hour day and nine-month year 
that are used to serve non-participating students. To the extent that time allocated to 
Chapter 1 is taken away from other instruction, the presumed additional instruction 
provided by Chapter 1 may not be an addition at all but merely a replacement of one 
form of instruction for another,^ though presumably these new services are designed 
to be more intensive than the regular services. Second, opportunities to learn can be 
greatly increased by increasing the efficiency of regular classroom instruction and by 
keeping students engaged in learning. Because Chapter 1 services are supplementary. 
Chapter 1 can do little to improve the regular Instructional program, and may in fact 
introduce inefficiencies as students are grouped and re-grouped, or as they move back 
and forth from their regular program to their Chapter 1 program. 

This need not be a violation of the supplement-hot-suppiant rule, for that 
rule refers to fiscal rather than educational relationships among programs. 
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Even if Ghaptcr 1 provided superior teachers to those that students otherwise 
rcccivc-and there is ho reason to believe that this would be so-the quality of 
Chapter 1 services may not be sufficiently better to compensate for poor teaching 
during the remainder of the day. Regardless of the quality of services provided during 
the Chapter 1 portion of the day, students still spend five and a half hours a day in 
the regular program, and the quality of that program wiH infiuence their ability to 
learn. 

The supplemental character of Chapter 1 is also implicated by the research 
evidence regarding the importance of schoblwidc goals and program coherence. To the 
extent that the supplemental nature of Chapter 1 creates a separate cadre of teachers 
with duties, goals and responsibilities different from those the regular teachers have 
toward Chapter 1 students. Chapter 1 may diminish the coherence of the total 
educational program that students receive. The research findings suggest that 
Chapter 1 teachers and services should not be isolated from the rest of the school 
program-some have even suggested that the legislation should not restrict the use of 
Chapter 1 funds to supplemental services. Without such a restriction, the argument 
goes^ funds could be used to improve the regular program. But it is also true that, 
without such a restriction, students may lose benefits, such as smaller student-teacher 
ratios^ they now receive from Chapter 1. 

Another feature of Chapter 1 for which these findings are relevant is its 
provision for parent involvement. Chapter Vs predecessor. Title I of the ESEA, 
required parent advisory councils and provided considerable detail regarding the nature 
and role of these councils. These provisions were curtailed in Chapter 1. While 
jparent involvement in the governaiice of local projects may serve other important 
goals, the research reviewed here suggests that parents* direct involvement in their 
own children's education is more likely to have a positive effect on student 
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achievement. Such iiivblveirient occurs when parents encourage their children to do 

their homework, when they attend school and observe classes, and when they tutor 

tiieif children, for instance. 

But such involvement may be difficult to accomplish through Chapter 1 for two 

reasons. First, Chapter 1 constitutes only a portion of the child's school day, and it 
is not considered to be the main educational program. Traditional mechanisms for 
obtaining parent involvement, such as report cards or parent-teacher conferences may 
not be sufficiently flexible to incorporate Chapter 1 instruction. Second, to the extent 
that Chapter i parents are themselves less educated, they may feel intimidated by 
educational institutions and be reluctant to participate in their children's education. 
Further^ the quality of parent-school relations may vary considerably from district to 
district, so that no general rule of thumb would necessarily apply. Therefore, the 
implications of these findings for determining whether or how the Chapter 1 legislation 
could facilitate this form of parent involvement is not clear. 

Developing an optimal role for Chapter 1 to play in the variety of schools in 
which it now exists is extremely difficult. Yet it is clear that this role, and its 
relationship to the regular program, makes a difference to students. Schools may try 
to coordinate Chapter 1 and the regular program by using common tests or common 
textbooks, by placing Chapter 1 teachers or aides under the supervision of regular 
teachers, by maintaining joint record-keeping systems on student progress, and so 
forth. The variety of practices is probably so great that Chapter 1 services sometimes 
enhance the rest of the school program, sometimes hinder it, and sometimes exist 
beside it but completely independent of it. 

Without more knowledge about what actually happens in Chapter 1 classrooms and 
in regular classrooms, we cannot know which of these research findings-or 
cdmbinations of findings-best explain the achievement data we have reviewed here, or 
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which suggest the most fmitful directions for increasing the effectiveness of Chapter 1 
programs. The National Assessment of Chapter 1 is currently completing several 
studies designed to describe the services currently provided under Chapter 1, including 
examinations of how these services relate to the regular pr gram, whether they are 
consistent with research on educational practices for disadvantaged students, and the 
extent to which they are influenced By Federal as opposed to State or local policies. 
These findings will be presented to Congress in our final report. 
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NATIONAL ASSESSMENT OF COMPEN^ATXmY EDUCATION 
ASSISTED UNDER THIS CHAPTER 



"Sec. 559 (a) The Secretary shall conduct a national assessment of compensatory 
education assisted under this chapter, through independent studies and analysis by the 
National Institute of Education: The assessment shall include descriptions and 
assessments of the impact of (1) services delivered, (2) recipients of services, 
(3) background and training of teachers and staff, (4) allocation of funds (to school 
sites), (5) coordination with other programs, (6) effectiveness of programs on student's 
basic and higher order academic skills, school attendance, and future education, and 
(7) a national profile of the way in which local educational agencies implement 
activities described under section 556(b). The National Institute of Education shall 
consult with the Committee on Labor and Human Resources of the Senate and the 
Committee oh Education and Labor of the House of Representatives in the design and 
implementation of the assessment required by this section. The National Institute of 
Education shall report to Congress the preliminary results of the assessment required 
by this section in January and July of 1986, and a final report shall be prepared and 
submitted to the Congress hot later thah Jahuary 1, 1987. 

"(b) Notwithstahdihg ahy other provisioh of law or regulation, such reports shall 
not be subject to ahy review outside of the Department of Education before their 
transmittal to the Cohgress, but the Presideht ahd the Secretary may make such 
additional recommendations to the Cohgress with respect to the assessment as they 
deem appropriate. 
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ADNfiNlSTRATlVE STATUS REPORT 



The National Assessment of Chapter I was legislated as part of the Technical 
Amendments to the Education Consolidation and Improvement Act of 1981. Enacted in 
December 1983, these amendments required the National Institute of Education (NIE) to 
provide Congress with a final report by January 1987, just three years later. The 
major milestones for this National Assessment, either accomplished or projected, are 
listed in Table 1. The remainder of this administrative status report reviews each 
aspect of the administration of the National Assessment of Chapter 1. 

Milestones 

Following passage of the legislation, NIE hired a Study Director to design the 
National Assessment of Chapter 1 and to oversee its implementation. The Study 
Director joined NIE in April 1984. 

During the summer of 1984, the Study Director discussed the study's purposes and 
Congress* information nce(?^ with several Congressional staff members; with Department 
of Education staff, both within the Chapter 1 program administration and in the 
bepartnient's Office of Planning^ Budgeting and Evaluation; with members of the Office 
of Management and Budget, the Congressional Research Service, the Congressional 
Budget Office^ and the General Accounting Office; with many educational associations 
and interest groups within the Washington area which were known to have an interest 
in the Chapter 1 program and its future; and with a variety of educational researchers 
and program evaluators. 

On the basis of these discussions, NIE developed a plan for the National 
Assessment of Chapter 1. The plan was reviewed by Department of Education officials 
in late summer and in October it was presented to Congressional staff. Two briefings 
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table 1 



Milestones for the National Assessment of 
the Chapter 1 Program 



December 1983 

April 1984 
October 1984 
December 1984 
May - September 1985 
October 1985 

January 1986 

July 1986 

January 1987 



Gongress passes legislation requiring the Natidiiai 
Institute of Education to conduct a National Assessment 
of the Chapter 1 program. 

National Institute of Education hires a Director to 
oversee the National Assessment. 

National institute of Education presents a Study Pian to 
the Congress. 

National institute of Education completes hiring a !5tudy 
Team to implement the Study Plan. 

National Institute of Education procures a series of 
independent studies for the National Assessment. 

National Institute of Education isj-eplaced by the Office 
of Research within the Office of Educational Research 
and Improvement. 

Of/Jce 9? Research produces its first Interim Report, as 
required by the Congress, for the National Assessment 
of Chapter 1. 

The Of f ice of^Research is scheduled to produce its 
Second Interim Report for the National Assessment of 
Chapter 1. 

The Office of Research is scheduled to produce its Final 
Report for the National Assessment of the Chapter 1 
program. 



B-2 



EKLC 



were neid, one lor Jienate staff and one for House staff. Following these briefings, 
further changes were made in the plan; 

In November 1984^ the final plan was submitted to Congress and NIE began in 
earnest to implement it. Concurrently, NIE began forming a Study Team to implement 
the plan. Qualified researchers both within the Department and outside it were 
solicited. A few Department staff began in the summer of 1984, but researchers from 
outside the Department did not join the Study Team until Decembfi. 

The Study Plan outlined a number of separate investigative components which, 
taken together, would provide information regarding the full range of questions and 
issues that had been raised during the preceding summer. The first stages of 
implementation of the Study Plan consisted primarily of contracting assistance in 
carrying out a number of these components. NIE chose to contract out portions of the 
work in part because the level of effort involved in doing these projects did not make 
in-housc work feasible, and in part because contracted studies assure a level of 
independence often necessary to give the overall study credibility. 

Requests for proposals for these studies were prepared throughout early 1985, 
advertised through the spring of the year, and contracts for the projects were awarded 
throughout the summer. The full list of procured studies appears in Appendix C. 

In October 1986, a year after NIE presented its Study Plan to Congress, the 
Secretary of Education reorganized the Office of Educational Research and 
Improvement (OERI) in such a way as to integrate NIE's components into the rest of 
the office. Under the reorganization, the Chapter 1 Study Team was located in the 
Office of Research (OR), one of the five components within the Office for Educational 
Research and Improvement. The Office of Research contains the research functions 
that had been previously placed in NIE, and consequently is the closest approximation 
to NIE that now exists. 
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This report constitutes the second of three reports which wiil be delivered to 
Congress as part of this National Assessment of Chapter 1: The jFinai Report will 
describe findings from all contracted studies, summarizing virtually every aspect of the 
current operation of Chapter 1 programs. 

B udget 

The budget for the National Assessment has proved to be one of its most 
complicated and problematic, features. This has occurred for three reasons. First, the 
study was expected to be funded from three sources, rather than one, thus requiring 
three separate budget lines rather than one. Second, one of those sources, the 
Chapter 1 budget, is forward-funded. This means that its budget does not normally 
become available until three-quarters of the way through the fiscal year. For fiscal 
year i98S, Chapter i funds could not contribute to the study until well into the fiscal 
year. For fiscal year 1986, the Department asked Congress to make a special provision 
in the Chapter 1 budget which would enable Chapter 1 funds being used for this 
N2.tiohal Assessment to become available earlier. The special legislation resulting from 
that request greatly facilitated the progress of the National Assessment. Finally, the 
third source of funds, the Secretary's Discretionary Fund^ was impounded by the 
Federal District court in ChicagbVand therefore was not available at the time or in 
the amount that the Chapter 1 study had anticipated. As a result of these budgetary 
difficulties, the Study Team solicited funds from programs within the Department of 
Education other than those specified by the Congress. NIE signed agreements with 
both the Office of Bilingual Education and the Off ice of Special Education such that 
these offices agreed to support studies of topics of mutual interest. However, also as 
a result of these budgetary difficulties, many of the procurements were postponed 
because funds were not available when they were originally anticipated. Finally, 
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virtually every project had to be IBcreraentally funded so that those fuSds that were 
available at first could be used to start as many projects as possible. 

table 2 summarizes the contnbutions made by each funding source to date; 



Table 2 



Contributions to the National Assessment of Chapter 1 
(in thousands) 



Funding Source FY '83 FY '84 FY '85 FY '86 total 

^*»^Pt«^ ' $376 $409 $1,500 $1,100 $3,376 

300 450 1.481 2,231 

Chapter 2 

(Secretary's Fund) gOQ 450 1.250 

Bilingual Education 35O 75 

Special Education • 200 266 

T^^'f^i^ $376 $700 $3,300 $3,106 $7,482 

No FY '87 funds have been appropriated for further analyses of the National 
Ass«sssment's data. 

Curren^Statys 

the current status of the National Assessment of Chapter 1 is as follows: 

• Most funded projects are nearing completion. Data were collected 
from school distrjcts during the 1985-86 school year. Data are 
being analyzed and final reports are being prepared. 

• The Final Repiort to Congress is being prepared. 

Despite difficulties in awarding contracts and getting contracted studies uHdcrway, 
OERI still plans to complete the National Assessment on schedule. However, the 
timeline is such that many interesting analyses will not be completed. The staff of the 
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National Assessment plans to present a series of separate special-tdpic reports in early 
1987 to enhance the findings presented in the main report Some topics being 
considered for special reports arc: services provided to private school students by 
Chapter I» the provision of Chapter 1 services to tEP students, the relationship 
between Chapter 1 and special education programs, and the role of States in providing 
compensatory education services. 
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NATIONAL SURVEYS/STUDIES 
A_ Study of fcocal im biementatipn of ECIA^Cfea^tei^ ^ 

eontractar: Research and Evaluation Associates, Inc., Washington, DC 

Subcontractor: Westat, Inc., Rockvillc, Maryland 

Project Officer: Ron Anson 

This survey of 2,600 school districts included question: about all the major 
provisions in the ieglslatlon: 

• parent involvement 

• program evaluation 

• needs assessment 

• selection of schools and students 

• services to private school students 

• program design and resource allocation 

• administration and record-keeping 

The survey sample was designed to partially overlap with the sample of school 
districts which participated in the 1980-81 District Practices Survey^ thus permitting 
cross-time comparison in these districts. 

In addition to the mailed questionnaire to 2,000 school districts, the survey 
included in-depth telephone interviews with 200 of the districts which responded to the 
mailed questionnaire. 

A Survey of Chabfet-l^ Schools and Teacherg 

Contractor: Westat, Inc., Rockville, Nfaryland 

Subcontractor: RMC Research Corp., Hampton, New Hampshire 

Praject Officer: Gil Garcia 

These researchers surveyed by telephone staff members in roughly 1,300 schools 
across the country. Principals of Chapter 1 schools were asked about the 
characteristics of their schools and about their Chapter 1 program configurations and 
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Chapter 1 teachers were asked about their education and experience and about the 
services they actually provide to students. For comparison purposes^ regular classroom 
teachers and teachers of other categorical programs were also sampied and asked 
analogous questions. 

The survey included: 

• Elementary Schools with Chapter 1 Programs 

- Some with high cdaceiitratidns of poverty.. 
Some with high concentrations of litxiited-English 
proficient Students. 

• Elementary Schools Without Chapter 1 Programs 

Some with other kinds of compensatory education. 
Some with no cqmpcnsatory education students but with 
disadvantaged stu^dcnts. 

Some witli no compensatory education and with very few 
disadvantaged students. 

• Private Elementary Schools with Chapter I Programs 

• Middle Schools and Secondary Schools 

- Sdihc with Chapter 1 Programs 

Some with other cbrnpensatory education programs 

A CES Fast Response Survey (FRS) of 
Chapter i Oversight 

Agency: Center for Educational Statistics 

Project Officer: Marty Orland 

This survey of 760 school districts contained questions about local experiences 

with State program monitoring, state audits. Federal management reviews and Federal 

audits. Questions were asked regarding the number of experiences districts have had 

with each type of oversight, the content covered by the oversight review and the 

nature of changes in practices, if any, that resulted. 
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School Stadents' Pai-tieipatioii Ut- GMBtei- 4 

Agency: Center for Educational Statistics 

Project Officer: Richard Jung 

this survey of 900 school districts jprbvided national information about the number 
of nonpublic school students iaarticipatihg in the Gfiapter i program as well as the 
location and time (e.g., before or after school) of such services. Data on these and 
related issues were collected for the 1986-87 school year and. retrospectively, for the 
1984-85 school year, in order to compare levels and types of Chapter 1 services for 
nonpublic school students before and after the Agullar v. Feitnp Supreme Court 
decision. In this decision, the court riiled that the provision of Chapter 1 instructional 
services in sectarian schools was unconstitutional. 

A^tudY of the Costs of Snemaj^ HueattQfl^ftrvi(p ff^. 
Amended to Include Costs ^ofU ^apt<i^^^Sefvie<> s 

Contractor: drc. Inc.. Washington. DC 

Project Officer: Nferty Orland 

This study, funded initially by the Education Department's Office of Special 
Education, was desigSed to determine the resources (and their costs) used to provide 
special education services under a variety of specific program arrangements. NIE 
amended the contract so that the investigators were able to document the resources 
used to provide services under a variety of Chapter 1 program arrangements as well. 
The Chapter 1 program options include elementary in-class programs, elementary pull- 
out programs, secondary programs, after school programs and so foith. The study was 
conducted in a nationally representative sample of 60 school districts. 



C-3 

122 

o 

ERIC 



the Nation al Assessment of Educatinnal 
Progress (NAEP^ 

Contractor: Educational Testing Service, Princeton, New Jersey 

Subcontractor: Wcstat, Inc., Rocfcvillc, Maryland 

Though the primary purpose of the NAEP is to document student achievement, it 

also includes data on students* school and home experiences, teacher and classroom 

characteristics and school characteristics. The National Assessment of Chapter 1 

designed ah amendment to the NAEP grant to conduct special analyses regarding 

Chapter 1 students and Chapter 1 schools. Data will not be available in time for our 

final report. 

The Title I Evaluation and Reporting 
System mERS^ 

This reporting system, developed in response to Title I regulation, continues to 
exist in a reduced form under Chapter 1. Local education agencies report to State 
agencies who in turn report to the U.S. Education Department. These reports include 
data regarding the number of students served by grade level and service provided. 
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CASE STimtFS 



Ar^tttd^or Admi nistration 

Contractor: Abt Associates, Inc., Cambridge, MA 

Subcontractors: Education Commission of the States, Denver, CO 

Policy Studies Associates; Washington, DC 

COSMOS, Washington, DC 
Project Officer: Marty Orland 

Investigators documented administrative practices in both State and local 
educational agencies during the 1985-86 school year and any major changes that 
occurred since Title 1. Topics included: 

• At the State level: 

Monitoring and enforcement 
Technical assistance 
Application approval 

Policies in areas where the Federal law has changed, 
such as parent involvement, comparability, and 
evaluation. 

• At the local level: 

Needs assessment and evaluation 
Program desigri decision-making 
Funds allocation policies and practices 
Parent involvement 
Application and reporting activities 

The study entailed visits to 20 States, with return visits to nine of them. Then, 
in each of the nine States, three school districts were visited. 

A Study of School Di strict Pr^r^ 
Design D ecision-M^ictnp; 

Coiitractdr: SRI International, Menlo Park, California 

Subpontra^lor: Policy Studies Associates, Washington. DC 

Project Officer: Ron Anson 

Investigators determined how districts and schools make prograS design decisions 
and compared districts and schools that changed their approaches to those that 
remained constant. The study had two goals: 
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• To gain a better uiidefstanding of why districts and schools 
change or maintain key features of their Chapter 1 programs. The 
study examined the influences of: 

i Legislative change from Ti^^ to Chapter i 
Shifts in State or jocaj policies 
Chajiges in budget contexts 
Program design preferences of State or district 
administrators and teachers 
Apprehension about Federal audits 
Instiiutionalizatioh of the Chapter 1 program 
Conviction that the program is successful and working 
well 

• To examine decisions to adopt or jforego particular program design 
features of current public interest. Examples of such features are: 



- Programs in secondary schools 
i In class program designs 

Reliance on aides vs. teachers 

Schoolwide projej:ts 

Chcn[yges in thcjntcnsity or grade levels of services 
Parent involvement 
The use of computers 
Emphasis on higher order skills 

The study was conducted in 20 school districts and 60 schools. 



A Study of How Bistricts Allocate Resources 
Among Schools 

Contractor: Educational Testing Service, Princeton, New Jersey 

Subcontractors: Gaffncy, Anspach, Schember, Klimaski and Marks, P.C., 

Washington, DC 

Decision Resources, inc., Washington, DC 
Project Officer: Marty Oriand 

Investigators examined both the decision-making processes used to allocate 
resources among schools and the resource distributions— the number of teachers, aides, 
or computers per child or per grade level that resulted from those decisions. 

Investigators: 

m Described the infiuence of State and Fcdcfal laws on local 
decisions. 

• Determined whether different decision-making strategies yield 
different patterns of resource allocation. 



• Showed the effect of df TEereht resource aliocatloh strategies on 
economically- and educatiohally-disadvantaged schools and 
students. 

• Petermined the effect of multiple-needs students, and of multipie 
Federal and state prograris on Chapter 1 resource ailocatioh 
patterns. 

• Described changes iii resource allocation from Title I to Chapter 1. 
Twenty districts in eight States were visited. Investigators interviewed both 

district and school staff regarding their decision making, and documented actual 
resources that resulted from those decisions. 



A Study bf^Tgettng Practices U s ed in the 
ChaDtei^4^rbgram 

Contractor: SRA Tcchnj)ldgics, Inc., Mountain View, California 

Subcontf actor: Northwest Regional Educational Laboratory, Portland, OR 

Project Officer: Dick Juiig 

Investigators examined the liet effects of Chapter 1 school and student selection 



procedures on the characteristics of the studeiits served in the program. The study 
analyzed data on student poverty level, achievemeiit status^ grade point average, 
attendance rates, grade retention patterns, and participation in other categorical 
programs for Chapter 1 and non-Chapter 1 students in 30 districts. The five major 
questions addressed were: 

Ho^w do districts determine which schools and students receive 
program services? 



What rationale(s) underlie district policies and practices for 
selecting project schools and participants? 

How do Chapter 1 schools and students differ from non-Chapter 1 
schools and students? 

Are different types of students served under Chapter 1 than were 
served under Title I? 

What are the effects of varying school and student selection 
practices on the characteristics of students served in the program? 
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Thirty districts were selected so that they wiere diverse with respect to size^ 
Urbaiiicity, region, poverty level, cdiicehtratidn of liihited-Ehglish-prdf icieht students^ 
and the presence or absence of non-Federal compensatc«y education programs. 

A Study of the Whdie-Dav Instructional 
Experiences of Chapter 1 Students 

Cdntractdr: Far West Educational Labdratory, San F'ranciscdi CA 

Project Officer: Randy Denialine/Gil Garcia 

Investigators described the actual configuratidn of services that Chapter 1 

students receive over the course of a school day and over the course of a school week. 

They determined: 

• Students* exposure to various instructional topics; 

• How services are coordinated across service providers. 

• The quality of instruction provided. 

• The services provided in the regular classrooms while Chapter 1 
students are pulled out. 

• What teachers and students perceive the role and purpose df 
Chapter 1 to be. 

The study included 24 schools distributed over six geographic regions. The 
schools encompassed elementary, middle, and secondary levels, and some were private 
schools. Within each school eight students were followed for a day and two were 
followed for a week. The students will vary in grade level and achievement levels, and 
in the cohfiguratioh of services provided to them. 
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COMMISSIONED PAPERS 

Effects of Alternative Besicns ii^ 
Compensatory Education 

Cbatractor: Research and Evaluation Associates^ Washington, DC 

Project Officer: Randy Dcmalinc/Ron Anson 

A number of independent scholars were asked to review research on the 

effectiveness of various program design features used in compensatory education. 

Features included the following: 

• Staffing patterns 

• Service configurations 

• Relationship between Chapter 1 and the regular program 

• Curriculum 

© Overall compensatory education strategies 

Once these research summaries were completed, other researchers were asked to 
critique them. All summaries and critiques were then reviewed by a panel of 
educators, and their findings have been bound together in a summary document. 

Alternative Strategjgg in Coiiwen satorv EdAfcat4<^ 

Contractor: Research and Evaluation Associates, Washington, DC 

Project Officer: Ron Anson 

A number of prominent authors and practitioners were asked to prepare papers on 
issues that policy makers may consider in the upcoming reauthorization of Chapter 1, 
ECIA. The issues related to: 

• The Federal role 

• Access and accountability 

• Program implementation 
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The papers were reviewed and critiqued by the authors and other researchers and 
practitioners. After responding to the critiques^ final versions of the papers will be 
bound in a summary document. 
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Analysis of School Bifstrict and St ate Educatimi 
Agency- Record;; 

Contractors: Seven contracts have been awarded to school districts oi 

State agencies. 
Project Officer: Gil Garcia 

State and local agencies analyzed their data bases on Chapter 1 students to 
answer questions regarding the coordination of Chapter 1 with other programs and 
regarding the effectiveness of Chapter 1 services. Two categories of studies were 
funded: 

• Investigations of the patterns of categorical services Chapter 1 
students receive over several years; and 

• Investigations of the long-term educational accomplishments of 
compensatory education program students. 

Each State or local agency conducted analyses that were appropriate to the 
particular data bases it had available. 

Data Analysis and-^eolmfeal^iitmnrt 

Contractor: DRC, Inc.. Washington, DC 

Subcontractor: Pblicjr Studies Associates, Washington, DC 

Project Officer: Bea Birmah 

This contract serves a number of purposes for the National Assessment of 
Chapter 1. The contractor is responsible for: 

• Creating a data library 

• Conducting computer analyses of large data bases (e.g.. Census, 
Sustaining Effects Study) 

• Conducting literature reviews and issue analyses 

• Coordinating data collection and analyses across all of the other 
procured studies 

These activities will provide information that the National Assessment will use in 
writing its three Congressionally mandated reports. This contract has already produced 
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Chapter 1 state profiles, ah overview of State compensatory education programs for the 
National Assessment of Chapter 1, and studies of Federal administration of the 
Chapter 1 program. 
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DESCRIPTIbN OF DATA SOURCES USED IN THIS REPORT 

This ajjpcndix describes the data sources used in the report and assesses their 
technical quality and applicability. The review of each data source presents 
background information and describes relevant technical aspects, including sampling 
procedures, student and program measures, and measures of outcomes or impacts. Each 
review then evaluates the strengths and weaknesses of the data source. 

The following data sources are reviewed: 

• The Instructional Dimensions Study (IDS); 

• the Sustaining Ef fects Study (SES); 

• The title I/Chapter 1 Evaluation and Reporting System (tiERS); 

• The National Assessment of Educational Progress (NAEP); 

• The CB0 Report on trends in Educational Achievement; 

• The Sustaining Achievement Study (SAS); and 

• the study entitled Summer Le arning and the Effects nf f^^hoAllnp 

Nffi In^TOctinnal Pjtnyp^ jons Study 

t Background and Study Characteristics 
A. Overview of the Study 

The IDS was one of several studies conducted by the National Institute of 
Education (NIE) as part of the mandate of the 1974 Educational Amendments to study 
compensatory education supported by ESEA Title I and by various States. Based on 
data collected in school year 1976-77 with follow-up data collected in the fall of 1977, 
the IDS sought to identify relationships between a variety of classroom structures arid 
processes and program effectiveness. The data for the study were collected from 400 
purposively selected first- arid third-grade classrooms, the IDS assessed program 
impacts on reading and mathematics achievement scores. 
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One major purpose of the study was to examine the differential effects of pullout 
and in class designs on the achievement of students who received compensatory 
education; The second major focus of the IDS was assessing the impact of several 
instructional practices and characteristics— including instructional time, individuali- 
zation, teacher-student interactions, classroom climate and motivation, and teacher 
training and experience— on program outcomes. 

B. Sampling Procedures 

HiE's approach to sampling for the IDS was to select classrooms that evidenced 
specific instructionai dimensions, without specifically seeking to produce nationaliy 
representative results; At the same time, NIE aimed at selecting a wide variety of 
classroom settings comparable to the diversity of the Title I program nationwide in 
1976. Therefore, classrooms were selected that varied oh student background factors, 
such as family income levels and ethnicity. In addition, classrooms were selected from 
school districts that varied in urban status, size, geographic location, and percent of 
Title I enrollment. 

NIE sought nominations from SEA and LEA officials, curriculum publishers, and 
lists of exemplary programs; Nominated districts and schools were then visited prior to 
final sample selection to ensure that practices of interest had actually been 
implemented and that those practices had been in place for at least one year. The 
final saniple was made up of 400 classrooms (200 first grade and 200 third grade) in 
100 schools from 14 districts. 

C. Student, Program, and Outcome Measures 

There is ho cvidchce in the report oh fihdihgs from the IDS (NIE, 1977) that 
backgrouhd data oh studehts were collected or used in the analysis; 

£:ctehsive data were collected on instructional practices employed in the 400 
ciassroohis studied by NIE. These data included (1) interviews with principals and 
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teachers to assess teacher backgrbuhd and experience, (2) ctirricuiar analysis to judge 
the overlap between curriculum and the content of the outcome measure (the 
Cdmprchchsivc Test of Basic Skills— CTBS), and (3) videotapes of each classrdbrn to 
provide detailed data on classroom characteristics such as teacher-student interactions 
and time on task. The IDS also collected information on costs associated with each 
reading and mathematics program. 

The study used the CTBS, level B, form S to measure achievement of first graders 
and CTBS level I, forms S (fall) and T (spring) for third-grade measurement. Students' 
attitude toward school was also measured in fall and spring. However, little analysis 
of this measure was reported except to relate in a footnote that fall measures of 
attitudes were high and remained high at the time of spring assessment. 

II. Strengths and Weaknesses 

The IDS has several strengths that make it relevant for any assessment of 
Chapter Vs effectiveness. Firsts it assesses the effectiveness of compensatory 
education practices. Some of the other studies reviewed in this technical appendix do 
not assess program impacts, or they focus on broader populations than just the educa- 
tionally disadvantaged. Second, the IDS collected the most cbmprehehsivc data on 
compensatory education practices for a comparatively large sample of classrooms. The 
study went beyond mean effects to examine the differential effects of pullout and in- 
class designs— the most common Chapter i instructional arrangements— and to evaluate 
finer grained classroom processes and characteristics such as time spent on specific 
learning tasks, the match of instruction to student needs, incentives provided for 
learning^ and teacher training and experience. Third, unlike TIERS, which contains 
data on effectiveness from whatever tests local school districts chose to use, the IDS 
used the same measure of achievement for all 400 classrooms. In addition, the study 
assessed impact on achievement gains of overlap between items in the spring test and 
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instruetiohal content CFor all groups of students, overlap between curriculum content 
and test items has a stronjg positive relationship to achievement" [p.32]). 

At the same time, the IDS has several weaknesses that limit the study's 
usefulness. First, the data are 10 years old. While data oh the efficacy of pullout and 
in-class designs and the impacts of classroom processes may still be relevant, the IDS 
obviously cannot assess whether changes in Title I/Chapter 1 since 1976 have 
influenced the effectiveness of the program. For example, other program models such 
as the replacement design and extended pullout programs were not specified in the 
Title I legal framework until 1978. the IDS tells us nothing about these approaches. 
Moveover, compensatory education is a more mature program, and staff have 10 years 
more experience on how to teach the educationally disadvantaged. Findings from the 
IDS and other studies-such as those about time on task arid the possible 
inappropriatcness of pullout designs for yourig childreri-have influenced the design of 
Chapter 1 programs, ten-year-old data can tell us nothing of the impact of these 
changes. 

The sample of the IDS also limits its usefulness, the authors of this study 
acknowledge that their aim was not to collect nationally representative effectiveness 
data, their sample was chosen purposivcly to ensure that practices of interest were 
included. Thus, their conclusions cannot be generalized beyond schools and programs 
in their sample. 

Finally, although the IDS apparently collected extensive details on classroom 
process, these data seem to have been underutilized. For example, the study reports 
larger average gains in raw scores, grade equivalents, and percentiles for those first 
graders who were mainstreamed compared to those who were in pullout programs. 
However, significance tests are not reported, and there is little assessment of the 
educational importance of these gains. With the wealth of data collected, it would 
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seem to have been possible to explore reasons why, for example, a pullout design was 
found to be less effective for first graders but more effective for third-grade 
mathematics programs. 

The Sustaihihg Effect^Stiidv 

L Background and Study Characteristics 
* A. Overview of the Study 

the S£S was a Federaiiy mandated investigation of compensatory education, 
including Title i and State and local programs. Conducted by System Development 
Corporation (SBC), the study examined the "nature, quantity, and environment" of 
compensatory education and "its sustained effects" (Hinckley, 1979, p. xvi). Funded at 
more than $26 million, the SES conducted a series of substudies including a longitudinal 
study, a cost/effectiveness study, a study of summer programs, a study of successful 
sites, and a participants' study. (Findings from some of these substudies were never 
published, however, because funding ran out.) The first year (1975-76) of the study 
was used ior planning, instrument development, and sample selection. Data were then 
collected in the fall of school year 1976-77 and in the next three school years. 
The SES addressed two major policy issues: 

• Who receives compensatory education? 

• How effective is compensatory education? 

In addition, the study examined several subsidiary issues: 

• What is compensatory education? 

• What is the nature of elementary school children's home 
ehvirohment, and how r j home environment related to school 
ehvirdhmeht? 

• What happens to studentsV achievement when compensatory 
education is discontinued? 

• What is the optimum duration and time for receiving compensatory 
education services? 
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^ ^^PP^J^^_JP achievement over the summer, and are summer 

school programs effective? 

B. Sampling Procedures 

The SES data and analyses were based on three samples: the Representative 
Sample (drawn to be representative of students in the nation's schools), the 
Comparative Sample (composed of 29 schools identified in high poverty areas but not 
receiving compensatory education resources), and the Nominated Sample (composed of 
43 Title I schools nomiiiated by Federal, State, and local officials as having particularly 
effective compensatory education practices). 

The Representative Sample is the main source of S£S data used in this report. It 
included 243 schools thnt were selected in the first stage of the sampling. These 
schools were randomly selected to represent schools stratified by region, size of 
district, and district poverty status. Schools from high-poverty districts were 
bversaihpled. Based on enrollment lists for fall 1976, the second stage sample was a 
systematic random sample of children enrolled in the schools. Certain subsets of 
schools and students were excluded from the sample: for example, schools that served 
mainly handicapped students and students in bilingual or £nglish-as-a-Second-Languagc 
programs. Because the study was never fully funded, the sample in follow-up years 
was reduced. The Representative Sample was reduced by 60 percent, and, overall, 
70,000 of the 120,000 students who participated in the first year of the study were 
included in the follow-up data collection. 

C. Student^ Program, and Outcome Measures 

The SES collected massive amounts of data on students, teachers, schools, 
compensatory education programs, and school districts. Student outcomes in reading 
and mathematics were measured with the CTES. In addition, students were 
administered a functional literacy test and an attitudinal survey of their views of 
school and of reading and mathematics. 
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For each year of data coilection, teachers completed a questionHaire with sections 
bh characteristics of the school and the teacher's background and details about 
instructional programs including whether grouping was used and what and how 
instructional materials were employed. Principals completed questionnaires on their 
backgrounds and school characteristics. The superintendent and business officer 
provided data on the district snd on the Title I program. Homeroom teachers provided 
background information about students and tracked students' program participation and 
attendance. Finally, a school coordinator-paid by the study-maintained a 
Compensatory Education Roster, which classified students according to the type of 
compensatory education services thesy received. 

II. Strengths and Weaknesses 

The strengths and weaknesses of the SES can be approached from two 
perspectives: criticisms of the reports from the substudies and strengths and 
weaknesses of the data base, which is available for reanalysis. After a brief review of 
criticisms of the reports, the strengths and weaknesses of the data base will be 
examined. 

The SES draft and final reports have been the subject of considerable criticism. 
The report of the majority of a special panel that was convened to review the SES 
(Hanushek. Breneman. & Hauser. 1979) made the following general criticisms of the 
substudy reports: 

• the technical quality and exposition of the reports on the whole 
are below prevailing standards and in some cases are unacceptable; 

• Statistical analyses have serious flaws. 

• The structure and reporting of analyses limit or even preclude 
their usefulness for evaluating or developing future compensatory 
education policies. 
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Other critics were less harsfii Linn (1979), for example, who was also a member 
of the review committee, did hot concur with the majority report and submitted a 
separate memorandum with his comments. He concluded in part that ''although...the 
reports have flaws, they also contain some important and potentially enlightening 
ihfbrmatibh....[E]veh the two weaker reports [#3 and #4] provide considerable 
information about the characteristics of participants in [compensatory education]^ 
especially Title T (p. 2). 

Even severe critics admit that the SES data base has strengths. The majority 
review report praises the first year sample and notes that those data arc useful for 
describing who participates in compensatory education. That report also lauds the 
development, administration, and scaling of achievement tests used in the SES. Linn 
(1979) points out that administration of tests by the contractor is a particular strength 
of the study and that "the norming and vertical scaling of the achievement tests are 
about as good as could be hoped for" (p. 1). 

At the same time, critics have raised important problems with the SES data that 
must be considered when either examining substudy results Or reanalyzing the SES 
data. The following are the most serious problems with the data base: 

• The attrition of jschoois and students from the longitudinal sample, 
especially as it limits the geheralizabiiity of data to the national 
population; and 

• The absence or poor quality of data for measuring variables, other 
than student achievement, heeded to assess the effectiveness of 
compensatory educatioh, c.g., studcht background, student status in 
compensatory programs, and the characteristics of such programs. 

Reanalyses of the SES data base heed to adjust for these shortcomings, where 
possible. 
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The Title T/Chabter l-Evfl44)fltion and R e porting System ittERS^ 

I. Background and Study Characteristics 
A. Overview of the Study 

The TIERS was mandated by Congress as part of the Education Amendments of 
1974. As developed by the U.S. Office of Education, TIERS had several features 
including models for evaluating Title I programs, technical assistance centers (f ACs) to 
aid in the development and implementation of these models. State and local reporting 
requirements, and dissemination of exemplary Title I practices to states and local 
school districts. 

In 1981, ECIA Chapter 1 repealed the authorization for TIERS as part of an 
effort to reduce Federally imposed burden, paperwork, and supervision. Mandatory data 
collection and reporting of evaluation data occurred m the last three years of Title I 
(1979-80, 1980-81, and 1981-82). Since the enactment of EClA, collection, analysis, and 
reporting based on TIERS have been voluntary. The most recent published analyses of 
TIERS data are based on data from the 1983-84 school year (Carpenter & Hopper, 
1985). 

B. Sampling Procedures 

Unlike sOme studies discussed in this appendix, TIERS does not depend on 
sampling techniques to estimate achievement gains attributable to title I. (However, 
states can sample districts for reporting purposes.) Instead, it is a reporting system 
that aggregates data on gains from the project level to the district and then to the 
state level. States analyze and summarize district data to prepare reports to ED. 
These data-aggregated at various levels-are intended to assess how much additional 
gains in basic achievement levels were the result of Title I projects. 
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C: Student, Program, and Outcome Measures 

TIERS collects demographic data on program participants and certain descriptive 
information (e.g., the number of students participating in Title I, the amount of parent 
iiivblveineht in Title I projects, characteristics of Title i staff, and services that 
Title I provides io students.) TIERS also reports gains in achievement in the basic 
skills of reading, mathematics, and language arts. Selection of achievement tests is 
left to the discretion of the States and local school districts to help ensure that the 
tests used are closely matched with the goals and content of local projects. Changes 
in achievement arc reported in Nbrmal Gurve Equivalents (NCEs). Each NCE value ha 
a corresponding percentile rank so that, for example, "an NCE of lb corresponds to a 
percentile rank of 23" and is interpreted as "23 percent of the norm group scores 
below ah NCE of 10" (Linn, 1982, p.9). 

TIERS provides three basic models (Models A, B, and 6) for analyzing Title I 
achievement gains. Since almost every district uses Model A (the norm-referenced 
model), discussion of the strengths and weaknesses of the TIERS data will be restricted 
to that model. As its name implies. Model A uses test publishers* norms to estimate 
what students would have achieved if they had hot received Title I services. Model A 
"simply assumed that, in the absence of participation in a Title I project, participants 
would maintain, on the average, a constant position relative to the norm" (Linn, 1982, 
p. 10). 

II. Strengths and Weaknesses 

TIERS represents a useful attempt to provide school districts, states, and the 
federal government with data on the impact of the Title I program. Unfortunately, as 
critics have shown, there arc several reasons for viewing TIERS data skeptically. 

Statistical and other technical shortcomings include the following: 
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• Regression cfTccts: A^ A was designed to reduce bias 
resulting fro^m regression effects, even the developers of the 
evalaation models (RMC Research Corporationj admit that the 
regression effect is not completely eliminated and results in 
overestimates of about 1 NGE; 

• Inapplicability of test norms: Critics of TIERS have noted that the 
use of test norms that are developed cross-sectionaily may produce 
a modest positive bias (1 NCE) when compared to longitudinal 
measures of program gains. In addition, although test norms 
ideally arc based on representative samples of students, in practice 
the refusal of districts to participate in test normihg studies make 
test norms unrepresentative. According to Lihh (1982), "What 
remains unclear is whether the differences due to 
nonrcpresentativeness are consistent across publishers, and, if so, 
whether the bias this introduces inflates or deflates the estimates 
of gains" (p.l3). 

• Selection biases: The proper implementation of Model A requires 
?H^t_^'*?_P*'^tcst not be used in any way to make selectijsn 
decisions. Linn (1982) found anecdotal evidence that this 
prohibition is sometimes violated. For example,_a student may be 
selected to receive Title I services based on a different test nr 
teacher judgment, but his scores on the subsequent Model. A 
pretest indicated that he does not need Title I services. Teachers 
or administrators might then make the logical educational decision 

the student from the program. There arc no cbm- 
prehensiye data on the frequency of this_ practice; however, even 
if the impact of such practices is small. Linn points out that "it 
adds to the cumulative bias in the direction of overestimating the 
impact of Title I" (p. 15). 

• Nqnrandom attrition: Students who leave compensatory education 
programs prior to the end of the school year usually arc not 
available for posttcsting. to the extent that attrition is not 
random, bias of undetermined direction results. For example, more 
able students may "graduate" from the program before year's end 
s^q that needier students can be served. If posttest scores are not 
obtained froni^ these presumably higher scoring students, estimates 
of gains wili be underestimated. 

In add: ^n, the foHowihg practical problems lead to biased results— usually 
overestimates: 

• Conversion errors: As test results are aggregated in the TIERS 
system^ errors can creep in, especially as teachers and 
administrators manually convert raw scores to percentiles and 
percentiles to NCEs. Investigation shows that errors are more 
likely to be in favor of increased (rather than decreased) growth, 
which can amount to a 1 NCE overestimate, on average. 
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• Test administratidn: Because it is important to teachers and 
administrators to show positive program results, pretests and 
posttests may hot be administered under th^e same conditions. 
Thus, subtle— ani sometimes not so subtle--behdihg of the rules 
can take place. For example, the teacher might encourage all 
studeats to take the j>retest but not ihake the maximum effort to 
ensure that low scoring students take the posttest The frequency 
of stich practices is unknown^ but their occurrence may add to 
overestimation problems of TIERS. 



# Practice effects: A fclated soufce of positive bias is the extent to 
which gains are simply^ measures of improved test taking skills or 
successful teac^hing to the test. While it is important that t^c 
test selected^ for nacasuring Title I gains should be closel^matched 
with the content andjobjectivcs of J project (a poorly matched 
test would underestimaje program performance). Linn (1982) points 
out '*that some inflation of gains is to be expected as the result 
of learning that is specific to the test" (p. 19). 

Linn points out that many of these sources of overestimate are cumulative and 

"probably account for most of the relatively large estimated gains obtained for the bulk 

of the 1979-80 results which were based oh a fall-to-sprihg testing pattern" (p.20). 

Although fall-to-spring testing was the predominant pattern in the early TIERS data, 

the 1983-84 data presented in this report are based on annual tests. While annual data 

still are subject to overestimates resulting from, for example, regression effects, the 

estimates of Title I gains based on this testing cycle are more likely to be a true 

picture of the impact of Title I. 



TIERS data have other weaknesses, although they would not have a systematic 
effect on estimates of program outcomes. First, because participation in the TIERS 



system is voluntary, data arc not available from all States. In 1984-85, 41 of 53 SEAs 
reported Statewide data in a format that could be aggregated at the national level 
(Reisner and Marks, forthcoming). Second, because States are allowed to sample 
districts^ data often come from a subset of districts in a State. The small number of 
districts or questionable sampling methods in some States could result in 



nonrepresentative samples. Finally, the quality of data submitted by some States have 
been poor. Nevertheless, the achievement patterns reported by TIERS are consistent 
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from year to year suggesting that these pfoblems of data quality do not systematically 
bias estimates of achievement upward or downward. 

The National As sessment of Educationai Rr^gre^s 

I. Background and Study efiaracterlstics 

A. Overview of the Study 

The NAEP-which was originally sponsored by the Carnegie Cofpofation-is a 
Federally funded effort to assess national achievement in reading, mathematics, science, 
social studies, and other learning areas. The central goal of NAEP is to assess 
periodically the educational achievement of the nation's youth and changes in 
achievement. Begun In 1969, NAEP periodically collects cross-sectional data on the 
achievement of 9, 13, and i7-year-olds and young adults between the ages of 25 and 
35. 

B. Sampling Procedures 

NAEP data arc derived from a multi-stage sample. The Primary Sampling Units 
(PSUs) have been stratified by region. State, community size, and socioeconomic status 
for the smallest size categories. Schools are randomly sampled within PSUs, and 10 to 
35 students are randomly sampled from selected schools. 

NAEP sample sizes arc large. For example, roughly 29,06(3 9-year-olds, 41,000 13- 
year-olds, and 36,000 17-ycar-olds were involved In the 1978 assessment of reading. 

C. Student^ Program, and Outcome Measures 

Since the major goal of NAEP has been to assess educational progress, the major 
data collection effort has been focused on achievement outcomes. Outcome data have 
been collected in 10 areas: art; career and occupational development; citizenship- 
literature; mathematics; music; reading; science; soclai studies; and writing. As of 1983, 
all of these areas had been assessed at least once. Citizenship, social studies, art, and 
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music had been assessed twice. Reading, writing, mathematics, and science had been 
assessed three times. 

Learning objectives are developed and revised over time in consultation with 
members of the educational community. Test items and exercises are written to assess 
these objectives. Tests and other measurement instruments are administered by NAEP 
staff (the contract for conducting NAEP was previously held by the Education 
Commission of the States and is currently held by the Educational Testing Service) to 
minimize burden on school district and provide for as nearly standardized test 
administration procedures as possible. Although new test items and exercises are added 
for each reassessment, common items are retained to permit analysis of changes in 
achievement over time. 

NAEP collects limited data on student background variables. The following scaled 
background variables have been collected: parent education, hours spent on homework, 
parent's occupation, presence or absence of items in the hbme^ gender, and 
race/ethnicity. No information is available through NAEP on program variables. 

11. Strengths and Weaknesses 

One major strength of the NAEP is the quality of the achievement data collected. 
Care is exercised to identify learning objectives, match test items to these objectives, 
and administer assessment instruments in a consistent manner. At the same time, since 
test items are not and cannot be matched to specific curricula and teaching methods in 
the nation's schools, underestimates of achievement levels and changes may result. 

Another strength is the sampling design for this data base. However, it is 
important to note that some groups arc systematically excluded, which probably results 
in overestimates of achievement. First, only students in school are eligible for 
selection. Thus the 17-ycar-old sample excludes dropouts. In addition, handicapped 
students and those with limited English proficiency are excluded. Finally, school 
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districts can, at their discretion, exclude students who cannot be properly assessed; 
These limitations in the NAEP samples have impiicatioh for the usefulness of these 
data for evaluating Chapter 1 since the excluded groups overlap with the population 
targeted to receive Chapter 1 services. 

Another shortcoming of the NAEP data for assessing Chapter 1 is that they do 
hot indicate which students receive compensatory education services or what the 
characteristics of such services are. 

Finally, as Koretz (1986) points out, NAEP data are not particularly useful for 
analyzing trends in test score data. One reason for this problentt is that there has 
been no formal equating of scores from one assessment to the next. In addition, NAEP 
has traditionally reported results in terms of percents of students correctly answering 
an item. Finally, NAEP usually does not report standard deviations. 

The CBO Report oh Trends in Educatm nal Achi evement 

I. Background and Study Characteristics 
A. Overview of the Study 

Trends In EdygatiOPal^AglHgvgmg at is a report prepared by the Congressional 
Budget Office (CBO) at the request of Congress. (Daniel Koretz of the CBO had lead 
responsibility for the report.) the report documents and investigates the decline and 
subsequent rise in trends of national educational achievement. The study is based on 
rcanalyses of existing data from the Scholastic Achievement Test (SAT), the American 
College Testing Program (ACT), NAEP, the National Longitudinal Study of the High 
School Class of 1972 (NfcS), the High School and Beyond study (HSB), and annual test 
data from the State of Iowa. 

The report discusses: topics in aggregate trends (e.g., when the decline began, 
how long the decline has lasted, what the depth of the decline was, and wheh the 
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decline was reversed) and differential trends among groups of the school-age population 
(e.g.^ differences in age group trends, in trends for boys and girls, in trends for high 
and low achievers, and in trends for minority groups and whites). 

B. Sampling Procedures 

Since data for this report came from existing sources, sample characteristics must 
be linked to the original sources-either testing programs or studies. Some of these 
sources derived data from nationally representative or nearly representative samples. 
(For example, see the discussion of the NAEP sample, discussed previously.) Other 
data sources clearly have nonrepresentative samples. For example, the SAT and the 
ACT testing programs provide data on subsets of the schoo!-aged population that seek 
college entrance. The Iowa Statewide test data-although representative of that State's 
students-are not nationaiiy representative. 

C. Student, Program, and Outcome Measures 

Since this is a study of achievement trends, data sources were selected that 
included student achievement data over time. In some cases, such as the SAT data and 
the Iowa State data, trend information is available over decades. Other sources— such 
as the NAEP— have only two or three measurement points, which make them less useful 
for trend analysis. Although all data sources for the study provide some measure of 
students' educational achievement^ the details of what is measured vary because the 
data sources arc employed for different purposes. The SAT, for example, is aimed at 
predicting college success, and its developers see it as less appropriate as an 
achievement measure than the ACT. 

Data sources vary in the amount of student background data collected. Some 
sources (such as the NtS and HSB) have fairly extensive background data; others such 
as the SAT and the Iowa data have little or no information on test takers. 
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Program data from these sources are limited to distinctidnSi for examplei between 
students who attend public or private schools or who attend schools in disadvantaged 
communities. 



11. Strengths and Weaknesses 

The report is ah excellent summary of national achievement trends; but, because 
it relies solely on existing data sources, its strengths and weaknesses depend in the 
end oh the strehgths and weaknesses of those sources. The report acknowledges those 
strengths and w<^aknesses and i3 circumspect about findings and conclusions on less 
than optimal data. The following summarizess acknowledged weaknesses in the main 
data sources for the report (for strengths and weaknesses of the NAEP. see the 
discussion above): 

• The SAT: In additibh 16 the nbhrepresehtativehess the sample 
and the fact that the SAT is primarily designed to measure 
achievemeht, a problem with the use of these data for trend 
analysis is that^ although there arc many years of SAT results, 
there arc ihcbhsistencies in the results over time. For cxamj)le, 
tabulations for recent yearsjare based oh the most recent test 
that a given student took. Tabulatiohs of results from the mid 
1950s to mid 1960s cbntaih all test results of each test taker. 

# The ACT: This data sburce has the advahtage over the SAT that it 
is intended to measure achievemeht as well ass predict cbliege 
Success. Like the SAT* the ACT is sbmewhat ihcohsisieht over 
time, which is a disadvantage fbr trend analysis. 



• The NLS^nd^fSB: Both of these Studies provide extehsive 
background data on students. At the same time^ the utility of 
both the NLS and HSB is limited for trend ahalysjs because only a 
few measurement points are available. Evidence frbm bthcr 
sources su^ggests that achievement trends were nonJihear during 
thc^iDacframec of these of the NLS or HSB to 
examine these trends probably leads to analytic distortions. 

• The Iowa Statewid^Satft! The major strength of these data for 
examihihg achievement trends Is that they provide three decades 
of annual equated test results^ However, their uscfulM^ is 
limited because, as already mehtiohed, the Iowa student population 
is not representative of studehts hatiohwide. in addition, the 
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Iowa data do not provide sufficient information oh student 
backgrounds to examine differential achievement trends across 
subgroups of students. 

The Sustaining Achievement Study 

1. Background and Study Characteristics 
A. Overview of the Study 

The SAS was conducted by staff of the Chapter 1 Technical Assistance Centers, 
which arc funded by ED to provide assistance to states and LEAs in conducting 
Chapter 1 evaluations and other technical activities. The overriding purpose of the 
study was to provide State and local evaluatbrs with information on how to implement 
evaluations of the sustained effects of Chapter 1^ as the Chapter 1 statute requires. 
The study used existing data to address the following questions: 

• Are there different patterns of achievement for students with 
varying patterns of participation in compensatory education? 

• Do achievement patterns vary across grade levels, between reading 
and mathematics programs^ and for subtest scores or specific test 
items? 

• Do achievement patterns differ depending ufion whether results arc 
based on fall-spring or spr ing-sprihg testing schedules? 

The data for the SAS came from three sources: (1) a review of locally conducted 
sustained effects evaluations, (2) a rcanalysis of test results from 17 school districts, 
and (3) rcanalysis of data from the Sustaining Effects Study, with particular attention 
to trends in individual test items, the review of the SAS will focus on the second 
data source and its rcanalysis. 

B. Sampling Procedures 

Data for the rcanalysis of local test data came from 17 school districts that had 
the required data available and were wiiiing to participate. TAC staff collected and 
analyzed spring 1982 and spring 1983 data for approximately 66,500 students in 
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Chapter 1 schools. Tfic authors acknowledge that although "these systems were 
selected primarily for the availability of their information, and they did not constitute 
a representative national sample, the overall achievement levels and participation 
percentages in Chapter 1 for students in the sample appeared quite similar to overall 
national figures" (Gabriel et al., 1985, p. 2) 

C. Student, Program, arid Outcome Measures 

The TAG staff limited test data to students in grades 2 through 6 since students 
in these grades make up roughly 70 percent of all Chapter 1 students. Test data were 
also limited to the two most used instruments in Chapter 1 programs: the California 
Achievement test and the Science Research Associates test. 

li. Strengths and Weaknesses 

The SAS is an example of the useful work that can be done with a limited budget 
using existing data, the strengths of the study are its documentation of the biases 
resulting from a fall-spririg testing cycle. (Sec the discussion of this problem in the 
foregoing dcscriptidri of the TIERS.) In addition the study identifies evidence of 
consistent longitudinal achicveriicnt patterns for four groups of students: those who 
never enter Chapter 1 ("out/but"), those who received Chapter 1 services one of the 
two years under consideration ("in/out" and "out/in"), and those who receive services 
both years ("in/in"). 

At the same tirtie^ the SAS-likc any rcanalysis of existing data -suffers from the 
weaknesses of those data^ which the authors acknowledge, in this instance, these 
problems result from the necessity of depending oh districts to cooperate and to supply 
the data they have. As already mentioned, this prohibits any generalization to 
Chapter 1 programs as a whole. Thus, although the SAS has produced findings that 
provide a useful perspective on the sustained effects of Chapter 1, the findings arc 
only speculative and require confirmation from nationally representative data. A related 
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prdbiem is that the data came from only li cooperative districts that haa data 
available, and over 75 percent of these data came from Inst foar districts, if one or 
two of these districts were unusual or had particularly flawed data, the geheralizability 
and quality of the data could be compromised even further. Because the TAC staff had 
to depend on locally collected data, they had no control over the quality of testing 
and other data collection procedures. Although they may have been able to control for 
some of these problems by limiting the number of tests considered, variability in local 
practices can have undetermined influences oh evaluation results. (See the discussion 
of TIERS for more detail on possible problems.) 

The need to examine longitudinal results also led to problems for the SAS. First, 
the study team was able to examine only two data points. Although their results are 
consistent across districts and across grade levels, even two or three more data points 
would help confirm and elaborate their trends. Second, as the authors admit, the 
requirement of two test scores for each student limits the result to the subsample of 
^tu^l^nts who were present for both tests. Given the limits of the data^ no estimate of 
the bias from student attrition is possible. 

Summer Learning and the Effects of Schooling 

I. Background and Study Characteristics 
A, Overview of the Study 

Undertaken in the early i97ds. Summer Le arning and the Effects of Schooling 
(Heyns, 1978) was a study aimed at disentangling the influences of socioeconomic status 
(SES) and schooling on achievement. The b^sic design of the study used summer 
vacation as a "natural experiment" to help determine the impact of social class when 
comparatively little organized schooling occurs. Heyns constructed a lohgitudinal data 
base from retrospective test scores for approximately 3,000 sixth and seventh graders 
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attending the Atlanta public schools; In addition, she collected questionnaire data from 
parents and information about summer experience. Her major finding contradicts much 
of the 1960s' and 1970s' literature on the effects of schooling: "Schooling makes a 
substantial contribution to cognitive growth::::Schooiing apparently attenuates the 
influence of socioeconomic status on achievement and thereby reduces the direct 
dependence of outcomes on family background** (p.l87). 
B. Sampling Procedure 

The data for the study were collected from a stratified random sample of sixth 
and seventh graders in Atlanta. Schools were stratified by race and by proportion of 
free lunch recipients (a measure of socioeconomic status). Within each stratum, 
schools were ranked by enrollment within each stratum and systematically selected from 
the strata. This resulted in a sample of 42 schools and a total sample size of 
approximately 4,800 students. The stratifying strategy resulted in a sample that was 
racially balanced and diverse regarding the SES of both blacks and whites. For further 
assurance of variability in SES, schools with high proportions of low income whites or 
with high income blacks were oversampled. 

The original sample was reduced by less than perfect response rates to the parent 
questionnaire and by students who did not have test scores for one or more of the 
three testing periods. The final sample of students— those with a valid parent 
questionnaire and three consecutive test scores— represented approximately 60 percent 
of the original sample (2,978 students). An additional 739 seventh graders had four 
consecutive test scores. Heyns chose to analyze these data separately in several 
instances. 

e. Background, Program, and Outcome Measures 

interviews with parents provided information on student background data. The 
response rate for the interviews was 86 percent. These interviews obtained information 
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bh standard indicators of socioeconomic status such as mother's and father's education^ 
occupational status^ income ievel^ and home ownership. These interviews were also the 
main source of program information— in this case^ information about summer school and 
other summer activities. 

The intermediate battery of the Metropolitan Achievement Test (MAT) was the 
outcome measure for this study. Because this was the districtwide achievenlent test 
for the Atlanta school system^ longitudinal outcomes could be obtained retrospectively 
and analyzed. Although students in the sample were sixth or seventh graders when the 
study was dbhe^ test scores for the sixth graders actually came from their fifth-grade 
fall and spring test dates and from the their sixth-grade fall test date (fall of 1971 
and spring and fall of 1972). Seventh graders' scores came from their fifth-grade 
spring test date (spring 1971) and from their sixth-grade fall and spring tests (fall 1971 
and spring 1972). For both cohorts, the summer period used for analysis was between 
fifth and sixth grade (summer of 1971 and summer of 1972). 

!!. Strengths and Weaknesses 

Overall, the quality of this study is very high. The study design and execution 
rest on a firm conceptual base, which is clearly articulated in the study. The sample 
design achieves variability on background variables of interest. One possible problem 
with the sample is the relatively high attrition rate of 40 percent. However^ the 
author shows that attrition appears to be randbm--at least with respect to race. (The 
final sample was made up of 60.6 percent of the whites in the original pool and 60.7 
percent of the blacks.) 

Care was taken in collecting student background data from parents. Just the fact 
that family income data and other SES indicators were collected from parents sets this 
study apart from other attempts to assess the effects of schooling. Further care was 
taken to attempt to interview in person parents who had no telephone, thus avoiding 
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an obvious bias in favor of students from higher SES backgrounds. Two other 
strengths of the study were its use of longitudinal data and its analysis olf data at the 
student level. Studies of school effectiveness often have relied on cross-sectional 
analysis and aggregated data. 

A final strength is that the study used extant data imaginatively to address 
several important policy issues. The longitudinal data base permitted creation of a 
natural control situation— summer vacation-to test the independent impact of schooling 
on cognitive growth. Employing test scores from a battery of tests, Heyns was able to 
examine the impact of schooling oh a number of subject areas, in addition, by 
collecting data on summer activities, she was able to explore in depth the nature of 
summer learning. 
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Over bne-half dozen measures have been used in this report to describe how well 
students did oh achievement tests. Vertical scale scores, NCEs, percentiles^ reading 
proficiency scores, and more have been cited to report whether one group of students 
learned more than another or how much a group of students has learned. All of these 
types of achievement scores are only dif fcicht ways of expressing a simple fact, that 
is, how many test items were answered correctly compared to some standard. 

Percchtiies 

Pcrcehtilcs arc the most commonly used scores for expressing how well a student 
or group of students has performed compared to other students. A percentile (some- 
times called a percentile rank) Is a score which expresses how well a given student or 
group of students has done compared to other students. For example, if a child's 
percentile is 40, we say the child did better than 40 percent of all other students on 
the test; that is^ 40 percent of the students taking that test got fewer of the items 
correct. But percentiles have some limitations, the most significant of which is that a 
percentile scale's intervals between points are not of equal size; that is, a percentile 
scale is riot ari equal interval scale. For example, it generally takes more of an 
achievement gain to improve from the 20th percentile to the 3bth thari it takes to 
improve frorii the 40th to the 50th simply because of the properties of the scale. 
Further, because the percentile scale Is not an equal interval scale, it is riot correct to 
use percentiles in many types of arithmetic operations, such as is necessary to 
calculate averages. 

Normal Curve^miivalerits 

Since averages across students' scores arc generally needed for program evalua- 
tions, an alternative to percentiles called the Normal Gurvc Equivalent (NCE) was 
developed. With NCEs, the students' scores could be averaged (permitting large scale 
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evaluation)^ they could be converted easily from percentiles to NCEs and back 
(permitting simplified reporting), aiid they used a lOO-point scale with an average of SO 
(permitting an easier grasp dri the "meaning" of the scale). The NCE scale was 
subsequently mandated for use in TIERS and is still in use under Chapter 1: 

As with percentiles, it is assumed that NCE scores will remain the same for a 
student or group of students in the absence of specific interventions. That is, all 
students will learn enough in the normal course of events to maintain their positions 
relative to one another, which means they are expected to get the same NCE score 
each subsequent time they take a test. An NCE gain or loss reflects a change in 
position relative to other students. 

Standard Scores 

Numerous other scales exist or can be created to express how well students did 
on a test compared to other students which assume that students' relative positions 
will remain constant despite the fact that learning has occurred. (It should be noted 
that no assumption of "equal" learning is involved; in fact, a child scoring at a higher 
percentile rank or NCE score is probably going to learn more in an absolute sense 
between tests than a child who originally scored lower; all that counts with these 
measures is relative position.) These measures fall within a family which is often 
given the general label of standard scores. 

All standard scores, of which NCEs are merely one example, are based on known 
qualities of the normal ("bell^shaped") curve. E>ifferent scores in this family result 
jfrom multiplying the standard deviation of the raw scores by different constants and 
adding different constants to the mean of the raw scores. Standard deviations measure 
the extent to which the scores of different children are dispersed around the mean or 
average score. If the scores are distributed normally, i.e., they describe a bell-shaped 
curve, then by definition a little more than 68 percent of the scores will fall within 
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one standard deviation above or below the mean, another 27 percent will fall within 
the second standard deviation, and a little less than S percent of the scores will fall 
further out from the mean than the second standard deviation. 

Based on the same raw score and depending on which mean and standard devia- 
tion "transformations" are used, one can report scores in terms of standard deviation 
units (the number of standard deviations, e.g., 1.25, above or below the mean), Z 
scores (with a mean of zero and a standard deviation of 1), t scores with a mean of 50 
and a standard deviation of 16), or NCE scores (mean of 50, standard deviation of 
21.06). The different scores arc only the result of different transformations; with a 
little arithmetic, any member of this family can be converted Into another. 

Figure E.l illustrates the relationships among percentiles, NCEs, standard deviation 
units, and the normal curve. The figure shows, for example, that ah NCE of 29 
corresponds to a percentile rank of about 16 and lies about one standard deviation 
below the standardized mean of 0 (NCE mean of 50). Standard scores provide a means 
of reporting the relative position of test-takers; they say nothing directly about how 
much more one student or group of students knows than another. To report those 
findings requires a member of the family of expanded scale scores. 

Exbahdc d-Scale Scores 

Students arc expected to gain knowledge between one test and the next just as 
older students, all other things being equal, are expected to know more than younger 
ones. Expanded scale scores have been developed to allow reporting how much more 
one student or group of students knows than another or how much a student or group 
has learned from one test to the next. 
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RgUre E.1 
Comparison of l^st Scores 
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Reading Prpf iciencv Scores 

NAEP's reading proficiency scores tlie veriical scale score used in the Sustaining 
Effects Study, and the expanded standard scores reported by NIB are examples used in 
this report which are members of the expanded scale score family. 

The expanded scales are developed generally as part of test-norming procedures 
and often involve administering some test items to students in grades above and below 
the grade level for which the items were designed. This permits subsequent estima- 
tions of the likelihood of marking the iir^^ correctly. These likelihoods can, in turn, 
be used to construct a common linear scale which cuts across test levels and which Is 
composed of scores which approximate an equal interval scale. Students' performances 
on the test, regardless of the level of the test they took, can then be placed on this 
scale to determine how much they know. As the children learn more, their position on 
the scale gets higher, reflecting their greater knowledge. 

If the expanded scale can be linked to some agreed-upon standard, such as the 
reading skill levels used by NAEP to describe the practical meaning of their reading 
proficiency scores (e.g., rudimentary^ basic, intermediate^ adept, advanced), then 
knowing how much furthcf up the scale a student or group of students has moved 
provides extremely useful information on prog.'am success. If, however^ no external 
linkage to an agreed-upon standari Is available, then judging the amount of achieve- 
ment growth also requires looking at comparis<;ns with other students. 

Grade^Etitjivalent Scores 

Grade equivalent scorc^' -vd been vfte most common measures ^nv reporting 
performance on standardized t^j.jis until the last few years. Grade ccji"^tvalents assume 
that levels of knowledge can he o' fcribe--^ . i erms of equivalent amo^i.^ics of time in 
school. For example, a fifth >rsd*. ^< - . r of t: .chcol year doino * average" 
work would be expected to obt^m a- i v '^veF ^ of 5.0, while a r t.deht in 
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the same class who ihight hot be doing as well may score "below leveP at, say, 4.2. 
Grade equivalent scales are not equal interval scales so they cannot be used with 
confidence to report how much more one student or group knows than another. 
Further, despite the logical sound of a statement such as, "The third graders were 
reading one and one-half years below grade level," the sense disappears oh closer 
examination, in this case, for instance, it is very unlikely that students midway 
through the first grade would be able to take the test at all. In fact, the farther 
scores are from average (and students in ehaptcr 1 often score well below average), 
the greater the amount of distortion in meaning found in grade equivalents. 

Summary 

Different test scores ere designed to accomplish different purposes, and gene 
speaking, test scores Fall into one of two families. The first family is based dircctl 
on the properties of the normal distribution and is used to compare the relative 
positions of test takers. Examples of members cf this family used in this report arc 
percentiles and normal curve equivalents (NCEs). The second family is based on 
relationships across levels of a test which permit developing equal interval scales. 
With ah equal interval scale tying together the diffcrcht levels of a test, it becomes 
possible to report how much students have learned from ohe time to the next or how 
much more one group of students khows than another. Among members of this family 
cited in this report are reading proficiency scores and expanded standard scores. 
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This appcididix pfeseats the information contained in Figures 3;2, 3.3, 3.4, and 4.4 
expressed iii tcfffls of Normal Curve Equivalents (NCEs) rather than in terms of 
percentile ranks as they appear in the text; 

Percentile ranks indicate the percentage of students nationwide that obtained 
scores lower than that particular achievement Icvcl.^ A percentile rank of 50* for 
example, indicates the achievement level which 50 percent of all students scores below. 
Percentile ranks must be interpreted cautiously because they arc not based on an 
equal-interval scale. This iheans that a child who increases achievement by 10 
percentile points* from 45 to 55 for example, has not learned the same amount as a 
child who increases iTrom 85 to 95. Percentile ranks cannot therefore be used for 
averaging student achievement gains since each point of gain does not represent the 
same amount of learning. 

Another measure which is derived from percentile rankings, the Normal Curve 
Equivalent (NCE), is an equal-interval scale. As a result, NCEs legitimately can be 
summed and averaged to compare the gains of different students or groups of students. 

All of the percentile ranks in Figures 3.2, 3.3, 3.4, and 4.4 in the body of the 
report were based on data calculated using NCEs or other standard scores. These 
scores were converted to percentile ranks for purposes of presentation. 



^The Chapter 1 Technical Assistance Centers (TAes) hav^^ that explain 

different types of measurement in terms suitable for lay audiences. 



F-1 



1B4 



Rglire 3.2 



Gains In NCE Scores for Chapter f Stuc^nts in 
Reading and Mathematics, 1983^ 



50 Reading 



AS 



40 



35 



30 



39£ 



386 




3&1 353 



34.7 34.7 34.7 343 343 



35S 



309 



32.0 



30.2 



305 



25 — 

2 3 4 

NCE 

50 iMathematic^ 



J l_J L-^-J l_J l_J I ii ii 



7 

Grade 



10 



11 



45 



40 



35 



44S 




41« 



39S 



360 



353 



3SB 



30 ; - 

25 



3e7 



3306 



=J ! J ! J L. , -J L I J L.J L.„-J I 

3 4 S 6 7 J 9 10 

Grada 



11 



Figure reads: From spring 1903 to spring ise the NCE ,??ores of ?«ccnd grade stadente who received 

Chapter I raadirrj instruction inc.'eased from 38.& to SO.fi, while the NCE scores of 12th grade 
students remained 'airiy constant around 29. 



Soaxe: M. A. Caipentor and P. A. Hopper, Symtwab of ChspOt I Dfita: Summaiy Mepof.. Re&on, \M: Advanced Technology. ise& 
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Qainsjn NCE Scorn for Title I Students In Reading and 
Mathematics, the Sustaining Effects Study, 1976-77* 
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Hgure roads: Frbm^thf^e fall to the spring iesiing, ist grad^ atudar^ts enrolled in Chapter 1 reading instructibri 
moved from just below ah NCE score of 39 to just atxive it. 

*NCE ^res presented are bas^id on scores from a yi-spring tecting cycle in contrast with the spring-spring 
cycle ased for TiiRS dsda in Figure a2. 



Source: M. Wang, M. Bear, J. CdnkJIh. R. Hodpfr.er; R^xyt 10: Compensatory Sendees and Educational DMiopment In ttw SctKtof Vtar. 
Santa Monica. CA: Systann Daiielopment Corp.. Iddl. 
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Gains in NCE Scores for Title 1 Students and Similcir Students 
Not Receiving Compensatory Education, the Sustaining Effects Sttidy, 1976-77* 
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Figure raids: From the Ml to the spring testirig,_lst grade Title I students receiving read 

fitgtl just tptow m NCE ^re of 39_to just abo\A9. while nefdy student in non-Trtle I schools 
dropped from just above an NCE score of 38 to just below a score of 3a 

^N6E soores f»Mahted ana b^ed on scores from a fall-spring testing cycle in contrast with the springFspring 
c^e used for TIERS data in Rgure 3.2. 



ScHiiw: ft WaoQ, I*. Biir, ttjepftm fleport 10: Compensahfy Services and akjcational DMidpment In the Schoot Vto: 

S«nta Monica. SyMom Dmiopnwnc Corp;. 1t61. 
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Comparison of Gains ih_NCE S^res iioiL Reading and 
Mathematics by lasting Cycie^ 1983^ 
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Figure reads: The HCE stoojvs of the a\^rage second grader receiving Chapter I reading instruction increased 
frorti 32-7 to 42S using a ^1 to spring testing cycje, while a spring to spring testing cycle resulted 
jn a jess dramatic isiCE score shift from 38.6 to 39.& 



Source: M. A. Carpenter arxJ P. A. Hopper, Syntfieste ot Chapter I Data: Summary Report Reston, VA: Advanced Technology, 19G5. 



F-5 



1R8 



ERIC 



TABLE F.1 

Normal Curve Equivalents <NCEs) and Perc«»ntite Ranks for Figures 3.2 and 4.4 
1983-84 Chapter 1 Reading and Mathematics Achievement Results 



Full Year 



Fall-Spring 



Reading 

Grade 2 
3 
4 
5 
6 
7 
6 
9^ 
10 
11 
12 

Hathematics 



NCEs 



38:^ 

35.: 

34.7 
34.3 
34.3 
30.9 
30.2 
28.9 



39.6 
38.2 
38.2 
37.8 
37.9 
37.2 
36.8 
35.9 
32.0 
30.5 
29.2 



Percentiles 



NCEs 



Percentiles 



Pretest Posttest Pretest Posttest Pretest Posttest Pretest Posttest 



29 

24 

23 
23 
23 
23 
18 



31 
29 
29 
28 
28 
27 
27 
25 
20 
18 
16 



32.7 

31.9 

32.4 

32.4 

32.5 

32.3 

32.4 

30.8 

30.6 

28.1 

27.2 



42.6 
40.0 
39.8 
39.0 
38.8 
37.9 
37.5 
36.1 
35.1 
32.2 
31.9 



21 
20 
20 
20 
20 
20 
20 
18 
18 
15 
14 



36 
32 
32 
30 
30 
28 
28 
26 
24 
20 
20 



Grade 2 
3 
4 
5 
6 
7 
8 
9 
10 
11 
12 



32.8 
32.5 
53.8 
33.!) 
33.8 
34.7 
34.3 
32.8 
34.1 
33.2 
33.9 



45.7 

43.3 

44.1 

42.2 

42.2 

41. 

40.5 

40.0 

38.3 

39.1 

38.4 



21 
20 
22 
22 
22 
23 
23 
21 
23 
21 
22 



42 
38 
39 
36 
36 
34 
32 
32 
29 
30 
29 



32.8 
32.5 
33.8 
33.5 
33.8 
34.7 
34.3 
32.8 
34.1 
33.2 
33.9 



45.7 
43.3 
44.1 
42.2 
42.2 
41.0 
40.3 
40.0 
38.3 
39.1 
38.4 



21 
20 
22 
22 
22 
23 
23 
21 
23 
21 
22 



42 
38 
39 
36 
36 
34 
32 
32 
29 
30 
29 



table reads: The percentile ranks corresponding to the second grade reading tines presented in Figures 3.2 and 4.4 
are 29 and 31 for the pretest and posttest, respectivety. The corresponding NCE scores are 38.6 and 



39.6. 



Source: M. Carpenter and P. Hopper, Synthesis of Chapter 1 Data: 
Technology, 1985. 
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TABLE F.2 

Normal Curve Equivalents (NGEs) and Percentile Ranks 
for Figures 3.3 and 3.4 

Fall-to-Sprin^ Readings and Matltematics Achievement 
Scores foj- Title I Stjidehts and Similar Students Not 
Receiving Compensatory Education 



NCEs 



Percentile Ranks 



Reading 
Grade 1 



Pretest 



Posttest 



: i 


title I Students 


38.8 


39.2 




Needy Students 


38.2 


35.9 


2 


Title i Students 


36.2 


36.7 




Needy Students 


35.2 


34.0 


3 


Title I Students 


34.2 


35.5 




Needy Students 


34.5 


34.0 


4 


Title I Students 


33.4 


34.2 




Needy Students 


35.1 


35.6 


5 


Title I Students 


33.6 


33.2 




Needy Students 


32.6 


32.4 


6 


title I Students 


32.3 


33.1 




Needy Students 


33.7 


34.0 



Grade 1 


Title I Students 


39.4 


41.2 




Needy Students 


38.3 


35.9 


2 


Title I Students 


37.4 


38.4 




Needy Students 


36.2 


35.5 


3 


Title I Students 


36.5 


39.0 




Needy Students 


35.4 


34.9 


4 


Title I Students 


34.7 


36.3 




Needy Students 


35.5 


35.8 


5 


Title I Students 


34.6 


36.3 




Needy Students 


34.6 


35.0 


6 


Title I Students 


36.1 


37.7 




Needy Students 


35.5 


35.1 



Pretest 



29.7 

28.8 

25.6 

24.1 

22.6 

23.1 

21.5 

23.9 

21.0 

20.4 

20.1 

21.9 



30.8 
290 
27.5 
25.6 
26.1 
24.4 
23.4 
24.5 
23.2 
23.3 
25.5 
24.6 



Posttest 



30.4 

25.2 

26.4 

22.4 

24.6 

22.4 

22.7 

24.7 

21.2 

20.2 

21.1 

22.4 



33.8 
25.1 
29.1 
24.5 
30.1 
23.7 
25.7 
25.0 
25.8 
23.8 
28.0 
24.0 



Table reads: 



The percentile ranks cbrrespbhding to first grade Title I reading 
students' lines in Figures 3.J and 3.4 arc £9.7 and 30.4 for the '^retest 
and posttest, respectively. The corresponding NCE scores are j8.8 and 
39.2. 

Source: M. Wang, M. Bear, J. Cbnkiih, R. Hbepfne?, Report i o- gSmbensatorv 

Sgrvices and Educational De^elo offlent in the School Year Santii Wonica, 
CA: System Development Corp., 1981. 
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TABLE G.l 

Sample Sizes for Figures 3.1, 3.2, 3.3, 3.4, 4.2 and 4.3 and table 4.1 



jPade/Subject 



1 Reading 
Hath 



Figure 3.22/ Figure 3.32/ 
Figure 3.3 Figure 3.4 



Figure 3.11/ Figure 3.4 



Figure 4.3 
(Spririg- 
Sprihg 
Testing 



(Title I 
Students 
in Title I 
Schools) 



2,785 
1,5^ 



(Need 
C(»f>eh3atory 
Education in 
Hoh-Title I 

Schools) 



table 4.15/ 



541 
1,051 



Figure 3.42/ 

(Son- 
Conpehsatory 
Students in 
title I 
schcbls) 



(Conpehsatory 
Education 
Students) 



(Non- 
compensatory 
Education 
Students) 



Figure 4.24/ 



(Not Figure 4.34, 
(Attended Attended (Fall- 
Summer Summer Spring 
School)* School)* Testing) 



2,115 
2,468 



344 
97 



296 
435 



2 Reading 
Hath 

3 Reading 
Hath 

4 Reading 
Hath 

5 Reading 
Hath 

6 Reading 
Hath 

7 Reading 
Hath 

8 Reading 
Hath 

9 Reading 
Hath 

10 Reading 
Hath 

11 Reading 
Math 

2 Reading 
Math 



93,959 
54^790 

115,160 
64,629 

119,437 
72^558 

121,383 
77,677 

105,021 
68,235 

65,246 
39; 072 

65,826 
45,842 

31,349 
22,635 

13 4?9 
8,372 

7,967 
5,096 

4,506 
3,352 



i^036 
n686 

2,392 
1,434 

2,227 
1,331 

1,982 
1,134 



678 
1,035 

591 
1,057 

6l3 
1,169 

548 
1,27b 

676 
1,231 



1,754 
2,207 

1,761 
2; 126 

1;876 
2;178 

1;992 
2^226 

1,992 
2,272 



512 
306 



305 
178 



324 
2C1 

371 
229 

498 
345 

241 

^5 
175 



1,013 
544 

1,136 
560 

1,132 

664 

972 
593 

831 
578 



172 



182,490 
63,922 

158,221 
68,215 

140,961 
68,328 

121,558 
63,350 

106,666 
55,456 

69,429 
36,483 

49,866 
28,589 

30,818 
18,012 

17,992 
7,485 

9,737 
3,297 

5,873 
1,859 
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Table reads: The grade 2 reading achicveinent values reported in Figures 3-1 and 4.3 for Spring to Spring t-sting are based on 93,959 students. 
* Grade at last test period. 

1/ H. A. Carpenter and P. A. Hopper, Synthesis of Chapter 1 Data: Sunmarv Report . Restbn, VA: Advanced Technology, 1985. 

2/ H. Wang^ H. Bear, J. Conklin, R. Hoepfrw^r, Report 10: Conpensatory Services and Educat tonal Development in the School Year . Santa Ronica, CA: 
System Development Corporation, 1981. 

3/ Natiorat IrstitQte of Ei^ation, Compensatory Edi^at4of^ Stikiv, Fih alr^eperV ^ Congr e ss f n'em^tie 4fBt4onftlr4r^kute-of Education . Uashir^ton, DC: 
September 1978. 

y t, Ktlbanoff and S. Haggapt- Report 8: SummeF^POH th and th e^ffet^vehess-of^ iinii e r Schbo lr . Santa Monica, CA: System Development Corporation, 



1981. 



to 
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Saipte Sizes for Achieveineht of Stucfents With 
tifferOTt Patterns of Particfpaf ioi^ fh Title I 
Across Three Years, Fi9ures 4.4 & 4-5 



Pa r ti c ipatio n Pattern Finure 4.4 

Y e a r 1 Year 2 Year 3 n^^^jyy. 



Participant 


Participant 


Participant 


140? 


Participant 


Participant 


N** ;-Participant 


680 


Participant 


Non*Participant 


Participant 


358 


Non- Participant 


Participant 


Participant 


7U3 


Participant 


Non-Participant 


Nbh-Participaht 


655 


Non*Participant 


Participant 


Nbh- Participant 


520 


Non-Participant 


Non-Participant 


Participant 


612 



Table reads: pr^ented in Fi^re 4.4 for three year participants, 

^^^^ P^''*'cipated in Title I in Years 1^ 2. and 3, 

uere based on a sample of 1407 students. 
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